A Look Into The Process Of Hiring Fresh Graduates At Arbisoft
Background
After struggling to hire the right engineering talent at the right time through our early years at Arbisoft, we decided that the only way to build a pipeline of such talent was to engage bright students even before they graduated, and lead them through a very rigorous recruitment process. Over the years this has evolved into one of the most comprehensive fresh graduate hiring programs in the country.
Hiring fresh graduates is a challenging but exciting process for the People Operations & Development (POD) team at Arbisoft. While fresh graduates may take some more time than experienced candidates to be productive members of team-Arbisoft, they usually don’t come with a baggage of behaviors from past experiences that can sometimes be hard to offload and unlearn. This is an extremely desirable trait.
The process is undoubtedly time-consuming and effort-intensive but the pay-off at the end is extremely rewarding. Some of Arbisoft’s strongest engineers were selected through this program and have honed their craft with us over the years to now be in positions to lead teams, train others and help with the refinement of the future iterations of the same program. These are not ordinary engineers. They are typically 1% of the pool of people we start evaluating through a process we are excited to open source today.
Our process
We have now been running the program for several years. Over time, we have attempted to refine our process by making it more rigorous, engaging and comprehensive. The purpose of this blog post is to highlight the diligence and effort that goes into selecting these talented young people who join Arbisoft as fresh graduates every year.
To kickstart this exhaustive but also time-sensitive process, we have put together a Fresh Hiring Advisory Team that oversees the process from beginning to end. This team has experts that contribute questions, design and implement the exam technology, and administer the different assessments required to be taken to qualify for a job at Arbisoft.
We start the process way before students graduate. Typically, in the last quarter of the year, we contact universities to gather data for students planning to graduate in the following year. Our attempt is to contact as many universities as possible across all of Pakistan. This usually ends up being 40-70 universities with under-graduate Computer Science or equivalent programs, from all four provinces as well as AJK.
We then evangelize Arbisoft as, what we believe to be, the best first computing job for fresh computer science graduates anywhere in Pakistan. We do that with the help of campus drives in which we encourage as many candidates to register for our first screening step.
Step 1 – the massive online exam
The first screening step in our process is a massive online Computer Science (GRE style) randomized and timed exam. The reason we are not concerned with large numbers of candidates at this stage is that the first exam is completely automated and fully online. It is based on our many many years of experience with building the Open edX platform – a course management system ideally suited to conducting large online exams. It is the same technology that powers edx.org, a platform visited by tens of millions of learners every day so we are extremely sure that the technology can handle many multiples of the exam load we expect.
The exam questions are chosen from a comprehensive library of questions contributed by other brilliant engineers at Arbisoft, many of whom have been hired through the same process in previous years. The questions on this first exam, test foundational topics in Computing like Algorithms, Data Structures, Graph Theory, Programming and Operating Systems. Topics that can be best learned through a formal Computer Science degree although a formal degree is not a requirement for appearing in this test or qualifying for a job at Arbisoft.
To protect the integrity of the exam, no single exam-question-contributor has any idea of what the final exam looks like. Not just that, each question’s values and answers are uniquely generated for each test-taker at the time of the test. So no two candidates, even if they are sitting next to each other, will find it useful to copy each other’s answers. This is done using a special question type named the ‘Custom Python-Evaluated Input Problems‘ in the edX platform. Also, every question requires the candidate to work out a solution and googling the question is of little or no help. On the contrary it is likely to waste precious time.
Step 2 – the job sample test
Top 10% of the candidates from this first test are then called to our premises for a full-day job sample test. The job-sample test is not timed and is designed to mimic a typical day at Arbisoft as closely as possible, while still maintaining the rigor of an assessment step. People have the entire day to attempt programming challenges that have complexity similar to the kinds of tasks we expect them to perform if we select them for a job at Arbisoft.
A regular day at Arbisoft
We try our best to make candidates as comfortable as possible and provide a memorable experience that includes meeting Arbisoft leadership and other senior team members, having lunch, and taking a tour of the facility. This stage spans a few weeks because we can only evaluate around 40 candidates per day.
Submissions from the job sample test are automatically tested using specialized grader technology that checks the submitted code for accuracy and completeness.
A final list of submissions, that pass this intermediate automated checking, is then reviewed by experts for the quality of code and the ingenuity of the solutions provided to shortlist a final set of candidates.
Step 3 – the behavioral interview
Successful candidates that pass the job sample test are then called in for behavioral interviews by our POD team. At this step, we are trying to make sure that there are no culture or commitment-related red flags for any of the selected candidates. Since this entire selection process is so demanding and expensive, we want to be sure that the people we end up hiring are willing to spend at least a few years with us and don’t have any values that clash with our core values of trust, collaboration, excellence, effort and value-creation. We are not necessarily looking for culture-fit, we are looking for culture-enrichment, and we want to guard against any clash in values.
People who do not make it through this final step are offered transparent guidance and advice on why Arbisoft may not be the best fit for them along with some possible next steps for their career progression.
People who pass step 3 are ready for our job offer. This is a very select group. Just to give you an idea, in 2020, 5000+ people registered for this process, nearly 3000 took the first exam, 400 took the job sample test and we made around 48 offers of which have nearly all have been accepted so far and 77% of them have joined us, as depicted in the funnel below.
Onboarding and training
We have a very specialized, accelerated training program for all of our fresh graduates. The details of this program merit a completely separate blog post but suffice to say that for the next 3-6 months after being hired, all these new joiners get intense training in the technologies we specialize in, shadow experienced engineers writing production quality code, and complete several mock projects of reasonable size to prepare them for production-quality code contributions. At the end of these 6 months nearly every single one of these graduates is ready to build an illustrious career in computing.
Challenges and iterative refinement
As with any new initiative, we have also had to face challenges. Four years into the program, we have ironed out many issues. Thus making the process leaner and better than what we started with. Some issues that still prevail are the biases that standardised tests may have, giving an advantage to certain cohorts. Dealing with scamming and hacking attempts is also a daunting task. While our team creates new problems and questions to cater to large audiences, making sure that no questions from previous years are repeated is a growing challenge each year.
These logistical challenges still need to be catered to but with regular retrospective meetings, we are able to come up with solutions and innovations to perform better every year. We welcome all ideas from internal and external sources to help us improve our process.
Statistics and numbers
When a relative analysis was done for two consecutive years about the performance of different universities, it was evident that candidates from LUMS performed better in both the years. Those from NUST, FAST and ITU followed suit and together they rank in the top 5 of the list of performances categorised by universities.
The following is a summarized result of the score distribution of various Universities’ candidates on our online test. The test covered areas including Programming, Data Structures, Algorithms, General Mental Ability, Discrete Math and Operating Systems, and as can be seen almost 80% of the people could only solve less than 50% of the test.
The chart below is a distribution of scores that shows a balanced normal curve for our online test scores. From a maximum score of 30, the highest score is 24 and less than 2% people managed to score more than 17. The median score was 9. Also, this chart shows how each of the universities performed overall. Universities whose color codes are concentrated more towards the right of this graph, did better than the ones that are concentrated towards the left.
Hiring new graduates is a process that comes with several different challenges. Arbisoft’s exhaustive hiring procedure, including testing, interview and on-job test, ensures that there is no mismatch between the company’s requirements and candidates’ skills and abilities; and that they are an organizational fit in terms of culture and flexibility.
