OpenSSL issues a bugfix for the previous bugfix – Naked Security | #linux | #linuxsecurity

If you’re an OpenSSL user, you’re probably aware of the most recent high-profile bugfix release, which came out back in March 2022.

That fix brought us OpenSSS 3.0.2 and 1.1.1n, updates for the two current fully-supported flavours of the product.

(There’s a legacy version, 1.0.2, but updates to that version are only available to customers paying for premium support, and given the changes and improvements in the product since the days of 1.0.2, we urge you to jump ahead to a mainstream version even – perhaps especially – if you plan to continue paying for support.)

The March 2022 update was a vital reminder that deeply-buried code with unusual bugs may end up getting overlooked for years, especially if that code is part of a complex, specialised, low-level function.

The bug fixed back then related to a special-purpose algorithm for computing what are known as modular square roots, which are more complicated to calculate than regular square roots.

Unfortunately, the code to perform this calculation, using an algorithm first discovered in the 1890s, was clumsily coded, tortuously written, poorly commented, and hard to follow.

However, given that it wasn’t in an obvious “externally-facing” part of OpenSSL, and given that rewriting it would have been a daunting task, we’re assuming that it was tested carefully for the correctness of its answers when presented with well-formed numbers, but not probed for its robustness when faced with unlikely input.

Because, when faced with digital certificates that had been booby-trapped to produce ill-formed numbers, OpenSSL’s BN_mod_sqrt() function could be tricked into looping forever, trying to close in on an answer that didn’t exist.

When you work only with integers, and disallow fractions of any sort, you find that many numbers don’t have modular square roots, just as you find that many integers don’t have regular square roots. Thus 7×7 = 49, so 49 has a square root that is a whole number, namely 7. But there’s no integer that can be multiplied by itself to give 50, or 51, because the next “perfect square” is 8×8 = 64. You can try for as long as you like, but you will never find a whole-number answer for √51.