[openssl.org #3024] Segfault with 'openssl enc' when using aes-cbc-hmac-sha1

[Steve Weis via RT]

OpenSSL 'enc' will fail with a segfault or other errors when attempting to
decrypt a file using aes-256-cbc-hmac-sha1. This behavior started with
recent Lucky 13 security patches and has been backported to older OpenSSL
versions, such as 1.0.1.

Apparently, 'enc' does not support the cbc-hmac-sha1 mode of operation.
This should probably fail immediately when used, rather than allow the user
to encrypt data, then fail on decryption. The same applies to gcm mode,
which is also not supported. Alternatively, 'enc' could support these
authenticated modes of operation, which would be a useful feature.

My version of OpenSSL and package:
--------------
$ openssl version
OpenSSL 1.0.1 14 Mar 2012
$ dpkg -l | grep openssl
ii openssl 1.0.1-4ubuntu5.8
Secure Socket Layer (SSL) binary and related cryptographic tools
$ md5sum /usr/bin/openssl
7902443a1f767a63e8656ccf6a6623e2 /usr/bin/openssl
--------------

Failing to decrypt a short plaintext:
--------------
$ echo "This is plaintext" > plaintext.txt
$ openssl enc -aes-256-cbc-hmac-sha1 -k password -in plaintext.txt -out
ciphertext.enc
$ openssl enc -d -aes-256-cbc-hmac-sha1 -k password -in ciphertext.enc
bad decrypt
140608068572832:error:0606506D:digital envelope
routines:EVP_DecryptFinal_ex:wrong final block length:evp_enc.c:532:
--------------

Segfaulting on an all-zeros input:
--------------
$ dd if=/dev/zero of=zeros bs=1 count=40
40+0 records in
40+0 records out
40 bytes (40 B) copied, 0.00196226 s, 20.4 kB/s
$ openssl enc -aes-256-cbc-hmac-sha1 -k password -in zeros -out
encrypted-zeros
$ openssl enc -d -aes-256-cbc-hmac-sha1 -k password -in encrypted-zeros
-out decrypted-zeros
Segmentation fault (core dumped)
--------------

GDB output of a version built from 1.0.2-dev:
--------------
(gdb) run enc -d -aes-256-cbc-hmac-sha1 -k password -in encrypted-zeros
-out decrypted-zeros
Starting program: /home/sweis/workspace/openssl/apps/openssl enc -d
-aes-256-cbc-hmac-sha1 -k password -in encrypted-zeros -out decrypted-zeros
...
Program received signal SIGSEGV, Segmentation fault.
sha1_block_data_order_avx () at sha1-x86_64.s:3394
3394 vmovdqu 32(%r9),%xmm2
--------------

Thank you.

[Rich Salz via RT]

The current releases warn about those modes not being supported.
--
Rich Salz, OpenSSL dev team; ***@openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org