Technical Assessment Of Developers

9 minute read

I’ve been looking for experienced Frontend Engineers and Full Stack Engineers for several months to join my team at NIST. I took this opportunity to explore how best to evaluate the professional qualifications of candidates. For some time I used different methods and compared the results. In the following, I will briefly present my results from this experiment and my future approach, which I have derived from it.

Presentation of a project of the applicant

Applicants are invited to present one of their previous projects. The presentation should not take more than 10 minutes. Subsequently, individual questions about software design, architecture and other aspects of the respective problem will be asked.

  • This method demonstrates very well whether a developer can communicate well with others. If he can identify the essential parts of a problem and present them clearly, this is very close to everyday work.
  • The approach is more suitable for experienced developers who have already implemented extensive projects or have contributed significantly to existing open source projects.
  • The time required for the preparation of the presentation is relatively low.
  • The willingness of the developers to prepare for this task is generally higher than to complete an artificial programming task that has nothing to do with the later working environment.
  • Even experienced developers do not always have projects that are in a presentable state. Especially for employees with family and other obligations such projects often fall by the wayside.
  • Candidates who are afraid of presentations and free speech have big problems with this type of test. They will probably give up during the preparation of the task in order not to have to face the situation.
  • For the interviewer, preparation is relatively time-consuming. In order to be able to ask meaningful questions, the interviewer should have examined the code before – if it is available in a public repository. If the code is not implemented in the normal working language, the effort increases again accordingly.

Coding Challenge

Candidates are given a programming task, which they have to solve within a certain time. Due to the limited time, the task will often not come from the everyday life of the company, but will solve a more or less arbitrarily chosen problem. The notorious FizzBuzz test is a common example of such a task.

  • This type of test is out of touch with reality and therefore hardly allows statements to be made about the candidate’s qualifications and suitability. The task also does not allow any conclusions to be drawn about essential areas of the candidate’s working methods. Unit tests, documentation, the use of software patterns etc. are usually not used under the time pressure of these tests.
  • The candidate is under enormous stress due to the time pressure. If the candidate’s laptop and his accustomed development environment are not used, the candidate also has to cope with an unusual system. If the candidate is still observed solving the task, the stress increases additionally. It is therefore easy to put even hardcore candidates under such pressure that they cannot generate a compilable line of code.
  • Experienced developers in particular are put off by this type of test. Every software developer who has significant professional experience should have proven that he is capable of solving such problems. Therefore they feel such tasks – probably rightly – as an insult.

Screening of Github repositories

Often developers have a Github account and include it in their CV. This is an invitation for the interviewer to have a look at the repositories of the account. If the repositories of the Github account are meaningful enough, the interviewer can refrain from further tests of the applicant’s abilities.

  • Junior developers who do not yet have extensive professional experience can use a public repository with the results of online courses, tutorials and challenges (such as Project Euler) to show that they not only enjoy tricky tasks, but are also willing to work on the required qualifications beyond their job.
  • It is easy to identify black sheep among the applicants. Anyone who publishes legally protected works, handles access tokens carelessly or, in the worst case, publishes their company’s source code disqualifies themselves immediately.
  • If a developer takes the trouble to maintain a repository, you get a good impression of how he works. This includes not only the quality of the source code and the corresponding tests, but also a good README and deposited license terms. One should keep in mind, however, that developers do not always show the same care in hobby projects as they do at work.
  • However, only few developers perceive a Github account in sufficient measure as an advertising platform for their own cause.
  • Many developers create dozens of Github repositories for small and unmaintained scripts. Checking such repositories doesn’t only take a lot of time when preparing the interview, it also doesn’t give any relevant information about the applicant.

Programming homework

With this type of test, the applicant is given a task in advance that requires several hours to solve. The candidate completes this task on his or her own computer with free time management. The results are then submitted by e-mail or simply deposited with Github. At NIST, we have prepared corresponding tasks for Frontend Engineers and Full Stack Engineers.

  • For junior developers who are not yet able to present existing projects, such a challenge is a good opportunity to demonstrate that they can solve an extensive task independently.
  • The task quickly shows whether an applicant is motivated and makes an effort to find a solution.
  • The test is not suitable for developers with more professional experience, since the task can only be relatively trivial due to the time required. Experienced developers therefore do not have enough opportunity to stand out from Juniors. This lowers the motivation of the developers and has a negative effect on their results.
  • The time expenditure is relatively large for the applicant. Developers with family and other commitments can be deterred from the effort.
  • The effort for the preparation of the interview is also large. If the code is not written in a language or on the basis of a framework that is familiar to the interviewer, the effort is even higher.

Analysis homework

The analysis homework has some similarity with the programming homework. The challenge here, however, is not writing code but analyzing a larger code base. The candidate gets access to a source code repository containing the code. This can be examined – also under free time management on his own computer. In a personal interview he is then asked for his opinion on the code.

  • In contrast to the programming homework, the complexity can be significantly higher here and thus does more justice to the self-image of senior developers. For junior and mid-level developers, however, such a task not useful.
  • It is easy to test understanding for common problems (e.g. when using design patterns) as well as for the use of different test and analysis methods. Points of interest for the interviewer may be whether the candidate uses tools for static code analysis or test coverage testing, or habitually uses UML diagrams to communicate about problems without much effort.
  • This type of test only achieves the best results if the applicant has at least rudimentary experience with the language and framework of the code to be analyzed. This either unnecessarily restricts the number of applicants or increases the effort involved in preparing several such tasks.
  • The fact that a presentation is created instead of code can become a major mental blocker for the developer. Compared to the programming homework this seems to be the case more often with the analysis homework.
  • The time expenditure is relatively large for the applicant again. Developers with family and other commitments can be deterred from the effort.
  • Also the preparation of a suitable task costs relatively much time and effort.

Pairing on some real world problem

The candidate is invited to the company for a period of at least a few hours and a maximum of a few days. During this time, the candidate works together with the developers of the team on a real task from the everyday life of the team.

  • Due to the long time and the close contact with the team, the team fit is ideally tested.
  • Also the technical knowledge as well as the soft skills of the candidate are tested extensively and in everyday situations. This enables a precise assessment of the candidate.
  • The candidate gets a comprehensive insight into the company, gets to know his colleagues, a typical task and the typical way of working (at least if the company also works in pairs).
  • The time expenditure for the company and the applicant is enormous.
  • Many applicants already see a homework assignment, which can be accomplished in a few hours, as a considerable investment with considerable risk. Applicants should therefore be paid accordingly for their time.
  • Such a test is legally problematic for applicants with an existing employment (at least in Germany). If the applicant takes time for it by taking a vacation at his present employer, he is obligated to recover and not put his manpower to use elsewhere.
  • Also the company inviting the applicant has to cope with various organisational difficulties concerning insurance protection, data protection, protection of trade secrets, payment of an expense allowance etc.

HackerRank

HackerRank is listed here as one of several similar online platforms designed to facilitate the assessment of applicants for developer positions. HackerRank offers two different approaches. Interested companies can use the platform to automate coding tests. Applicants can also work on challenges independently of concrete tests and thus prove their skills.

  • The candidate can perform these code tests on his or her own computer, and can also schedule the time relatively flexibly.
  • Automated coding tests are still coding tests – with all the disadvantages described above.
  • The second approach of the platform doesn’t work in my eyes either. Even after a few years of professional experience, a developer has probably worked with many different frameworks, libraries and languages. Hardly any senior developer will seriously try to prove all these experiences with time-consuming tests in HackerRank. A HackerRank profile will therefore only have a fraction of the significance of a Github account.

Conclusion

My conclusion after some months of testing and evaluating is unfortunately that there is no such thing as the optimal solution that can be applied to all applicants. However, I do not think that this is necessary either. At the beginning of this project in particular, I overestimated the value of a uniform test that would provide comparable results. It turned out to be much more important for me to find a test that suits the candidate and allows him to optimally present his strengths.

This brings us to the approach that I have been using since then to evaluate the technical skills of applicants. If an applicant mentions his Github account (or SourceForge, BitBucket, …) in the application, I will evaluate it after reviewing his CV. If the account is meaningful enough, I assess the candidate on the basis of this profile. If the account is not meaningful enough, I leave the choice between two options to the applicants:

  • Presentation of a project of the applicant
  • Programming homework (only for junior developers) or
  • Analysis homework (only for senior developers)

Updated: