| HOME | Software | Lotus Cars | DWARF | Kindle | Solar |
|
|
A new release January 2025 will have libdwarf fixes dealing with corrupt DWARF object files. These fixes seem a sufficient reason for release.
All known issues of libdwarf memory-access (cc -fsanitize etc, long run-time, crashes, etc. when reading fuzzed object files) found by the oss fuzz team (Created by Google) are fixed. Oss Fuzz changed the reporting from being email-based (early 2024) to being a list on a website (mid-2024) and I discovered the change in December 2024 (I guess we overlooked an email about the change). We finished fixing the dozen known issues today.
Now working on v0.11.2 Fixing oss-fuzz-discovered issues now I know how to see the unfixed items again. (Notification used to be via email, but that stopped.)
Actual release posted on github December 2 (but we call it December 1 anyway).
Revised handling of DWARF5 rnglists/loclists reading to be simpler, complete, and more accurate. Coverity Scan finds no issues. Will continue testing. Release of v0.11.1 expected in 3 days (December 1).
Added support for split-dwarf handling of .debug_loclists (with .debug_loclists.dwo). The code for .debug_loclists[.dwo] reading now follows the overall organization of .debug_rnglists[.dwo].
The library API does not change at all.
The libbacktrace github project uses libdwarf so folks using libbacktrace on latest MacOS DWARF5 debug information had trouble too. In libbacktrace project issue #123 I find mention of the same problem in May 2024! But I heard nothing about it till yesterday.
Thanks to myzhan (github) for reporting that section names new to DWARF5 were ignored making libdwarf useless and providing a small testcase.
Added 5 new name translations (example: __debug_names to .debug_names) and pushed to github.
libdwarf would report ranges incorrectly if one was reading an executable built using DWARF4 split-dwarf (a non-standard extension). There were coding mistakes in libdwarf, now fixed.
In addition, modern compilers producing DWARF5 split-dwarf are choosing to follow a single aspect of as-yet-unreleased DWARF6 and do not expect or want DW_AT_rnglists_base attributes in a DW_TAG_skeleton CU in .debug_info to be inherited by the correxponding DW_TAG_split_compile in the .debug_info.dwo. Thanks to Jeremy Rifkin for the error reports and a small testcase.
Several debugging-related functions have been added to libdwarf to aid in debugging split-dwarf related library code. These are not compiled except by defining special macros (undefined by default) and have no effect on the library API, but I've found them very helpful.
Pushed to github 16 Nov 2024 14:20 PST We are planning a December 1 2024 release.
libdwarf would report ranges incorrectly if one was reading an executable built using DWARF4 split-dwarf extensions.
libdwarf could crash if certain fields in DWARF5 .debug_rnglists or .debug_loclists sections were corrupted (say, by a fuzzer).
These are now fixed and I expect to push to github today.
We plan a release on December 1.
Pushed a substantial revision of the internals of dwarfdump counting uses of tags, attributes,and classes and forms when options like -kt -ku are used. Fewer malloc/free, less space used, many now-unused lines of code removed. There is no change in the library, just in dwarfdump.. This revision is a little smaller and faster than any previous version.
The statistics reports by dwarfdump -ku were always incomplete due to the lack of a portable tree-search tool at the time the reports were written. The -kt dwarfdump option was incomplete too (for many years). We have rewritten the record keeping using balanced search trees and reporting is now much simpler to write and more complete.
This is combined with fixes to Defects CoverityScan found (not previously reported) using the latest CoverityScan build tool. All pushed to github.
Pushed fixes addressing github issues from PolarNettles (github) and Jeremy Rifkin.
Thanks to PolarNettles for reporting a bug and providing a fine small objectfile so we could reproduce and fix the issue.
Thanks to Jermey Rifkin for finding a DWARF5 example that exposed a libdwarfbug and making it available.
Released 0.11.0 today.
And, the same day, I upgraded MacOS Sonoma with 14.6.1 and MacOS now deprecates use of sprintf, showing annoying messages that do not kill the build.
snprintf() appeared in C99. It appeared in The Open Group Single Unix Specification Version 2 (1997). Versions of libdwarf/dwarfdump after 0.11.0 will use snprintf in the single place we used sprintf. The change only causes trouble building with with an ancient libc but it seemed important to mention.
Am doing a last-minute addition of trivial casts to remove warnings from a build using latest MSVC. Places where we know the range of possible values. In 13 source files. The release of 0.11.0 will be done in the next day or two.
All the testing so far, on all supported platforms, passes its tests.
It is still impossible to get the regressiontests-code project to work on msys2: Windows file paths (C: or D:) sometimes show up and that prevents matching results in many many cases. The 200+ hour regressiontests runtime on msys2 is impossible to deal with. But make check in an msys2 build of libdwarf-code passes.
The readelfobj-code project was released at version 1.0.0. We fixed compiler warnings and fixed CoverityScan issue.
Now the code reports the DW_AT_name (if present) of the DIE referred to for all attributes with a FORM in the REFERENCE class. Much nicer result than the 31 July version. The code is even a bit simpler. So far my desktop little-endian Ubuntu and a big-endian Ubuntu machine pass all tests. Testing continues on FreeBSD32/64, MacOS, and Msys2.
Added code to dwarfdump to have DW_AT_specification DW_AT_abstract_origin DW_AT_type report the DW_AT_name (if present) of the DIE they refer to.
We plan to do a release of 0.11.0 around 15 August 2024 since we also now have better and simpler reporting of .debug_ranges data with the help of dwarf_get_ranges_baseaddress().
Thanks to Vincent Torri for correcting the calls to calloc() by placing the item count and item size in the correct places in the calls.
Some support for dwarf_get_ranges_baseaddress() involved changing CU reading and the call to dwarf_global_formrev_b() was ill advised during CU initialization. So calling an internal version that lets us avoid infinite recursion in case the DWARF is corrupt. So certain corrupt DWARF objects could lead to crashes in the library. See bug reports DW202407-011 and DW202407-012. Fix pushed today.
Added a new API function dwarf_get_ranges_baseaddress() to the API. Changed the version from 0.10.2 to 0.11.0 due to the added function.
The .debug_ranges section was defined for DWARF2, DWARF3, and DWARF4 and the definitions were identical across those standards. The range entries defined address ranges containing code. Libdwarf has always read and reported that section but the library documentation never clearly said that the offsets reported in a range were just that: offsets, not addresses.
To clarify the situation the documentation will, later today, specify these raw ranges are offsets and refer to a modified example which shows how to calculate the actual addresses. Dwarfdump is being modified to report both the raw (offsets) and cooked (address) data.
The changes involved are all in dwarfdump and code example examplev. Libdwarf is not changed.
The main branch is now 0.10.2.
Thanks to Jeremy Rifkin for again noticing a bug (rnglists, in split-dwarf, in this case) and providing the object files demonstrating the bug. For DWARF4 split-dwarf libdwarf now pays attention to DW_AT_GNU_ranges_base.
A bug leading to leakage when reading DWARF5 .debug_rnglists was only present for a few days and is now fixed. A bug where a corrupt .debug_rnglists section could cause libdwarf to malloc a large amount of memory is fixed, it too was only present for a few days. Both bugs introduced after 0.10.1 was released.
The handling of the GNU extension DWARF4 split-dwarf .debug_ranges section has been corrected.
The last instances of use of libc *sprintf() and its near relatives have been completely removed from the dwarfdump source by using existing features of dwarfdump's string functions on struct esb_s. Removing a few MSVC warnings. The function, esb_append_printf() was entirely deleted (it is local to dwarfdump) as no longer used.
New regression-tests based on ossfuzz issues have been added. All ossfuzz issues are fixed.
Release of 0.10.1 with the build/pkg-config problems in 0.10.0 fixed.
Release 0.10.0 is defective and is failing some people's builds due to improper libdwarf.pc (pkgconf) output. That omission mentioned yesterday here is the cause.
Due to an omission of one line from doc/Makefile.am doing a 'make dist' from the source in 0.10.0 and then building from that will fail in cmake builds. A 'second generation' problem. The correction to doc/Makefile.am is to add CMakeLists.txt as in the following diff:
EXTRA_DIST = \
Doxyfile \
+CMakeLists.txt \
mips_extensions.mm \
mips_extensions.pdf \
The correction is committed to 0.10.1, the tentative version of the next release (no date planned).
Released version 0.10.0 24 June 2024.
Now that we have reading of .debug_rntlists and .debug_loclists working properly for the first time it appears to be time to do a release. Due to the addition of one new API function (which is intended just for dwarfdump) the release id will be 0.10.0. Tentatively setting a release date of June 25, 2024. One commit message recently mentioned 0.19.0 but that was a typo. 0.10.0 is correct.
Thanks to Robert Mustacchi for pointing out a missing back tick from line 14 of scripts/run-all-tests.sh (reported in Issue #247).
Problems in reading DWARF5 .debug_loclists appear to have all been solved today. Fixes pushed to github. Regressiontests look good but are not 100 percent complete yet.
There were remaining problems in handling some situations with .debug_loclists and all have been fixed.
Thanks to Jeremy Rifkin for providing very useful object files with various uses of .debug_rnglists and .debug_loclists, they are now part of regressiontests.
TODO: remove some debugging $if 0 ...#endif sequences and add minor improvements to the printing of .debug_rnglists usage.
A push of substantial revisions reading .debug_loclists happened today. Revises the dwarfdump reporting of debug_loclists to clarify the dwarf 5 debug_loclists structure content. The --print-raw-loclists option is more useful now.
A push of substantial revisions reading .debug_rnglists happened today. Revises the dwarfdump reporting of debug_rnglists to clarify the dwarf 5 debug_rnglists structure content. The --print-raw-rnglists option is more useful now.
These changes also make it possible (depending on the size of .debug_rnglists/loclists!) to verify for oneself by inspection that DW_FORM_rnglistx DW_FORM_loclistx and their sections are valid.
On an older machine running FreeBSD 14.0 64bit tests of (big) objects with very large .debug_loclists and .debug_rnglists sections took over an hour instead of two minutes. And on newish machine -fsanitize on those objects takes 'forever'. Other cases/machines do not take an unusual amount of time. These suggest a libdwarf algorithm problem in reading those two newly modified sections. The library does not memoize more than a few values from those sections at this point -- others involve rereading section content repeatedly. I trust gprof/prof will help identify what is taking time --- very soon.
Thanks to Jeremy Rifkin for reporting diffulties reading a Split Dwarf Package File. There were errors in reading some .debug_rnglists content and DW_AT_ranges attributes generated an error. It's an entire set of steps to create files with this content, it not just compiling with a set of options. So it's not likely you would encounter this issue. The fixes have been pushed to github.
The same bugs as .debug_rnglists exist in .debug_loclists so arranging the same fixes in loclists/loclistx handling is under way.
Added support in dwarf_debug_addr_table() for reading DWARF4 GNU extension .debug_addr format data and printing it (with dwarfdump option --print-debug-addr ).
Thanks to kkraus24 (github) for noticing that in a cmake shared-library build we were not naming the .so files with the correct semantic version nor was the SONAME as it should have been all along. Thanks to flagarde (github) for fixing the cmake shared-library build.
Version number is now 0.9.3. As of today, no header file in the project does a #include of anything (no nested #include). The test sources libdwarf-code/test/test*.c had many violations of the project codingstyle as detected by dicheck, but all that is fixed now. Due to requirements for building libdwarf as a meson based subproject we now require meson version >= 0.54 . (Ubuntu 18.04 LTS meson is version 0.53.2).
Released version 0.9.2.
We plan to release 0.9.2 early in April 2024 (April 2, we think). Between now and the release we are reviewing the documentation to repair any errors or omissions. There are no known vulnerabilities in libdwarf. The only changes from 0.9.1 are fixes to vulnerabilities when reading corrupted DWARF. See https://www.prevanders.net/dwarfbug.html for details. Following the release we will be doing source housekeeping as the first changes in 0.9.3. For example, eliminating #include from all headers local to the project: such interferes with readability and the analysis tool clangd.
Another libdwarf crash was found by ifygecko. In this case a certain corruption in the .debug_loclists or .debug_rnglists section (DWARF 5) could result in a libdwarf crash. The simple fixes (checking for improper pointer values) have been pushed to github. DW202402-003. These bugs existed when the code was originally written for libdwarf-0.1.0.
A serious allocation error (which applies only to badly corrupted DWARF) results in accessing freed memory or data corruption internally to libdwarf, sometimes leading to SIGABRT or SIGSEGV. DW202402-002. Many thanks to ifygecko for finding these problems, for pointing out the outline of the solution, and for providing a set of small object files (created by the Hongg fuzzer) demonstrating the problems. The fixes have been pushed to github.
The problem was introduced when fixing an issue with small libdwarf memory leaks on certain corrupted object files and first appeared in libdwarf-0.1.0. The fix was...incomplete, in that it failed to consider all possible situations.
Thanks to rvijayc (github) for mentioning a problem with multi-gigabyte objectfiles under Windows. That provoked some refactoring and that fixed the problem.
Vulnerability DW202402-001, reported February 12, 2024, is now fixed (oss-fuzz 66646). A carefully crafted DWARF .debug_info could result in the library reading a single byte where it should not. The DW_FORM_ref1 code failed to check validity before dereference whereas all the other FORMs already did the check.
Following a 40 hour long loss of electrical power here at Libdwarf Manor (due to 'equipment failure' the power company says) the corrections are now pushed to github.
Having very recently eliminated compile-time warnings reported by MSVC, I tried VS code analysis, and though it reported many bogus things (suggesting use of windows-only functions instead of posix) it found eight actual bugs! Seven in dwarfdump and one in libdwarf. None of them reported by CoverityScan! Kudos to VS. Repairs in progress.
Pushed a set of changes that should allow libdwarf to read objectfiles larger than 2GB, even on Windows.
Thanks to Daniele Cono D'Elia and Matteo Marini for noticing (via fuzz testing) and reporting that fuzz/fuzz_crc.c results in libdwarf reading uninitialized memory. The fault is in fuzz/fuzz_crc.c as it fails to meet the requirements (we are updating the documentation to make the requirements clearer and fixing the test code bug).
Thanks to Vita Batrla for providing a patch to accomodate an i386 relocation type on DWARF data we had not seen used before (R_386_GOTOFF).
Thanks to TixoRebel (github) for bringing an inappropriate use of DW_FORM_ref_addr by an obsolete compiler -- we now let libdwarf work by detecting the compiler's producer string in DWARF, setting an internal flag, and altering libdwarf's behavior for applicable compilation units.
Thanks to rvijayc (github) for noticing, in the elf-reading code, cases where DW_DLV_ERROR was returned yet given the Elf documentation DW_DLV_NO_ENTRY is the proper return.
Thanks to Jeremy Rifkin for bringing some problems reading Macho relocatable object files to our attention.
We greatly appreciate their help.
The latest release, 0.9.0, was the first to require both zlib and libzstd to build libdwarf with decompression enabled. As a note on history, zstd was first released in October 2015, and by 2020 many standard Linux programs contained zstd-compressed sections to shrink executable size (Ubuntu 20.04, for example). While libzstd.so is present by default one may need (depending on the packaging of the release you use) to install the -dev version to have the headers that one needs to build libdwarf with decompression enabled. In Ubuntu version 20.04 and later that is libzstd-dev .
For large object files (executables, shared libraries) the developers use compression to shrink the size of DWARF debugging sections. There are two compression library shared object files in use: zlib (-lz,libz.so.1) which is installed by default in most distributions, and zstd (-lzstd, libzstd.so.1) which may not have the zstd.h header installed by default (but may have libzstd.so.1 installed by default). In Ubuntu 22.04 you will find libz.so and libzstd.so in /lib/x86_64-linux-gnu/
For most people this is a non-issue, most objects do not have compressed sections.
It's possible for most to see or create an object file
with compressed DWARF data (using zlib) with the
objcopy --compress-debug-sections
Release 0.9.0 can be compiled to read compressed objects if one has both libraries, but if one has only one of the libraries libdwarf will compile but will be unable to decompress object sections at all.
Given all this, folks creating releases with libdwarf have to make an unfortunate decision: compile libdwarf referencing libzstd.so which some will not have installed or build without either compression library.
Current source libdwarf (not released: 0.9.1) has options to build libdwarf without either compression library (see README.md, READMEcmake.md). By default 0.9.1 builds with references to zlib and zstd if and only if both of them are present.
The decision to force both libraries to be present to decompress zlib-compressed DWARF is for now awkward for some users. At some point both we be automatically available nearly everywhere, as zstd is simply better.
Small correct changes (mostly casts) with no practical impact on correctness have been made in 0.9.1 (versus 0.9.0). This has the effect of removing all relevant warning messages for those compiling with MicroSoft Visual Studio 2022 Community Edition. A few mild warnings from VS about using Posix functions are unimportant and will not be fixed. The Posix functions (read(), write() and others) are perfectly satisfactory as is. The dwarfdump '-O file=filename' option does not work, we hear, with dwarfdump built by VS. VS is not a supported environment, but if someone were to figure out a simple way to make 'dwarfdump -O file=filename' work properly with VS we would be happy to learn how it was done.
Released 0.9.0 on 8 December 2023.
It seems we are ready to do a release (of 0.9.0). Github actions and all regression tests are passing. Coverity Scan shows zero defects. Unless there is an unfortunate surprise the release will be tomorrow, 8 December 2023.
The improved cmake build
and
Thanks to Jeremy Rifkin and flagarde for improving the cmake build significantly.
Thanks to Nima Hamidi for carefully thinking about details of the API and revealing an error made in the API in 1992. Following the examples in the documentation using dwarf_next_cu_header_d() and dwarf_siblingof_b(dbg,NULL,is_info,...) works correctly so there is no practical problem. Calling dwarf_next_cu_header_d() twice before calling dwarf_siblingof_b(dbg,NULL,is_info,...) results in the CU DIE for the first CU header being inaccessible and possibly leading to the caller getting incorrect DIE data. The existing functions will be kept for several years at least.
The new functions are
libdwarf-regressiontests now work properly across freebsd,Ubuntu linux, and MacOS and in complete in well under an hour. Many tests pass on msys2 but not all, and the test run time on msys2 is very very long.
The goals for regression testing were to switch to meson for building and to do a better job with meson of finding libzstd and zlib (by finding such under /opt/local or /usr/local reliably). And to have regressiontests work on all supported systems. Regressiontests are working faster and more reliably with meson as the base compared to cmake or configure, though cmake and configure continue to work quite adequately for compiling and installing. After finally reading the entire mesonbuild.com document it was very easy to make the libzstd/zlib finding work right.
And if the web document shows black with white print (which I find unreadable) click on the little pair of sunglasses in the upper right hand corner.
Msys2/mingw64/windows regressiontests run so slowly that it is discouraging. Over 24 hours so far and only at 50% of the total testing regime. And passing mostly, but still...
The library and dwarfdump build fine and now pass their 'make check' sanity checks in all environments using any of the supported build systems.
Builds with meson will soon automatically find libz and libzstd even when those (and their headers) are under /opt/local or /var/local. The regressiontests use meson for their libdwarf build now and regressiontests shell scripts (if either libz or libzstd is absent) will skip the tests requiring libz or libzstd.
The last bit of configure here in regression testing was unnecessary and has been removed.
As of today the regressiontests do their libdwarf builds with meson instead of configure. We need gcc -fsanitize for testing, so now the meson builds allow that to happen. The last bit of configure there in regressiontests will shortly be removed. All the real work was done by sh or python scripting already.
As of 1700 PDT today there is a new test script in the source (scripts/allsimplebuilds.sh) It runs all the build mechanisms that the build platform is prepared to run, and skips where a build is impossible. For example, it checks for cmake, make, meson, ninja, and an msys2 environment (where appropriate) and where one of those is required but not usable the shell script skips that test. Tested on Linux (various) MacOS and Windows Msys2.
Now make check (or its equivalent for cmake or ninja) works in all supported invironments (see the examples in scripts/allsimplebuilds.sh for guidance).
The READMEwin-msys2.md has been updated significantly and is hopefully more useful about Msys2 than any earlier README.
The instructions for building on Windows MinGW64 had errors and were almost useless. Now there is a new file, READMEwin-msys2.md, in the source code with hopefully usable information.
libdwarf now supports reading Apple Universal Binary files. Thanks to Jeremy Rifkin for asking if we could support them. We added new functions in the API because modifications of existing API functions were needed to select a particular executable instance from an Apple Universal Binary to allow users/coders to select a particular executable for libdwarf to report on. So we'll rename the pre-release version 0.9.0 ,replacing 0.8.1. The pre-existing API remains in place unchanged, and it will automatically select execuable zero from the N in a Apple Universal Binary. dwarfdump gets a new option so one can select which executable from a Universal Binary to report on (dwarfdump defaults to executable zero).
Here is the detailed timeline:
2023-09-20 In github libdwarf-code discussion #185 Jeremy Rifkin asks if libdwarf supports Mach-O universal objects, which We will abbreviate here as Apple Universal Objects (AUB). Each AUB may have a number of Mach-O executables inside. 2023-09-21 Initial commit in the readelfobj-code project of code reading AUB. 2023-09-26 Completed code handling AUB in readelfobj-code project.. Most of this code will trivially port to libdwarf. 2023-09-27 In the libdwarf-code project we do a little code cleanup related to Mach-O object files. Added a new API function for AUB in libdwarf.h 2023-09-29 Initial commit in dwarfdump so dwarfdump can select what Mach-O object in an AUB to report on (defaults to object 0(zero)). 2023-09-29 Full AUB support is now public on github, 2023-09-30 DW202309-002 ossfuzz id 62833 DW202309-003 ossfuzz id 62834 DW202309-004 ossfuzz id 62842 reported. Bugs in AUB handling present publically for one day. 2023-10-01 Pushed fix for the three bugs in AUB handling: DW202309-002 ossfuzz id 62833 DW202309-003 ossfuzz id 62834 DW202309-004 ossfuzz id 62842 2023-10-02 DW202310-001 ossfuzz id 62943 reported on AUB handling. Present publically in libdwarf for two days. 2023-10-03 Pushed fix to ossfuzz 62943 DW202310-001
MacOS universal binaries are not properly handled at this point. Such has mulitiple executables in one executable object and one DWARF content set each set in a single object file. Recognizable by magic numbers.
Will this mean dwarf_init_path() must add a 'which version' integer for MacOS? Or maybe a new dwarf_init_version_a() with a new version number?
The tags for 0.8.0 got a little messed up:
v0.8.0 libdwarf-0.8.0
were created incorrectly (did not refer to a fully committed state) and have been deleted.
The correct tags are
v0.8.0-fixed libdwarf-0.8.0-fixed
Release 0.8.0 completed today. All release-internal dates are 20 Sepember, the release is just a couple days early.
We are considering making the next release 2.0.0 as a mark that we will not be breaking compatibilty in future.
The switch from date-versioning with 20210528 (ISO date) to today involved removing ancient interfaces (that did not work with DWARF5) and other mistakes. In addition, the source tree was completely reorganized to match current practices and essentially every source file was renamed for uniformity in naming. Many thanks to Vincent Torri for helping and advising though this entire process.
We will skip 1.x.x as gcc created version 1.0.0 in shared objects when we had date-versioning and we want anyone saving such a shared object be able to continue to use it.
We are on track to release 0.8.0 on or before 2023-09-20.
Running clang 14 -fsanitize on FreeBSD uncovered five issues. The issues: arithmetic on pointers or integers that (reading fuzzed object files) libdwarf was checking after the calculation. Now we do checking of values before calculation.
A pair of functions, dwarf_get_fde_info_for_reg3_b() and dwarf_get_fde_info_for_cfa_reg3_b(), have always returned a value as Dwarf_Unsigned through their dw_offset argument and that was always wrong. DWARF standards have always made it clear that is a signed value. So we added dwarf_get_fde_info_for_reg3_c() and dwarf_get_fde_info_for_cfa_reg3_c() to the API which have that argument correctly declared Dwarf_Signed. Adding functions is why we changed the release candidate version from 0.7.1 to 0.8.0. It's fine to keep using the original form as long as on realizes one must cast the Dwarf_Unsigned to Dwarf_Signed to use it properly.
Thanks to Osnat Levi for pointing out that is useful to add the O_CLOEXEC flag to each open() done by libdwarf (for environments that define O_CLOEXEC as available for open() calls).
Thanks to Zhiqin (Bill) Zhang for reporting erroneous output from libdwarf on an object file with relocation data but with the Elf type of ET_EXEC. Now we only do relocations on ET_REL object files. Fix pushed to github.
libdwarf has always handled UTF-8 and ASCII strings.
Dwarfdump now prints UTF-8 characters correctly if the build finds locale.h (defined in C90), langinfo.h, and nl_langinfo() are present and, at runtime, the system language code starts with UTF-8 (or UTF8) as returned by nl_langinfo(CODESET). If part of an individual string is invalid UTF-8 dwarfdump will use (on that string) the uri-style encoding it has always used (%xx format for each non-ASCII byte). One can also add --format-suppress-utf8 as a dwarfdump option to force all non-ASCII bytes to use uri-style. As always, dwarfdump allows --format-suppress-sanitize but that means all strings are passed to printf without replacing any problematic characters.
The version 0.7.1 release is planned to be on or about 20 September 2023.
The only known issue is that UTF-8 non-ASCII characters still print using URI-style as %xx for their bytes. See wikipedia on locale issues.
Problems compiling dwarfgen have occured in certain circumstances. From a fine suggestion from flagarde(github) dwarfgen/dg_getopt.c is renamed dwarfgen/dg_getopt.cc with a couple tiny alterations for c++ .
The UTF-8 work is stuck due to a configure problem not yet fixed.
Work done in the last few years on utf-8 checking makes it easy to quickly verify that a string will print well as UTF-8. Recall that the DWARF standard says UTF-8 is officially allowed in all strings.
At present any non-ASCII UTF-8 gets the (multi-byte) UTF-8 characters and ill-advised ASCII control characters transformed to %nn form, which makes such bytes ... hard to read.
Evaluation of certain details remains to be done before this can be put into dwarfdump.
Thanks to lingbin (github) to pointing out that cmake install was failing to install dwarf.h. libdwarf/CMakeLists.txt fixed now and pushed to github.
The code in src/bin/dwarfgen/dwarfgen.cc has been updated to write out elf itself. (No, it's not a full implementation of libelf writer code ... but it suffices.) The build scripts for configure/cmake/meson have been updated to remove libelf references and nothing does a #include of elf.h or libelf.h.
Which means nothing in the project now uses or refers to any elf.h or libelf.h or libelf. And the refressiontests are updated too.
Thanks to flararde(github) for suggesting this change and Vincent Torri for his encouragement and help.
The changes described here have been pushed to github release 0.7.1 . There are no logic changes in the library or dwarfdump.
There is no change for library callers. There is no incompatibility.
It's become clear that a few people running Windows want and need non-shared libdwarf builds under mingw-msys2. To that end we are changing meson builds to default to shared-library builds while adding the meson option "--default-library static" (which Linux/Macos users may choose to use). On Windows/mingw this creates a correct (only exported functions visible) libdwarf.dll It required a small change to libdwarf.h to make this work, but only changing #define lines (which become a bit clearer and simpler).
cmake continues to allow building non-shared libdwarf and one can choose shared explicitly with the following on the command line (even on mingw-msys2)
-DBUILD_SHARED:BOOL=TRUE -DBUILD_NON_SHARED:BOOL=FALSE
Autoconf continues to default to non-shared libdwarf and on Windows/mingw will only build if a shared-library build is requested.
New testing code verifies that the shared-library builds correctly hide library-internal functions from library callers for all three build mechanisms.
The essential idea of how to make all this work properly came from Vincent Torri along with a patch to get it started, and he deserves the credit. Thank you Vincent.
It's become clear that we have a historical mistake in:
typedef struct Dwarf_Regtable_Entry3_s typedef struct Dwarf_Regtable3_s dwarf_get_fde_info_for_reg3_b() dwarf_get_fde_info_for_cfa_reg3_b() dwarf_get_fde_info_for_all_regs3()
In the above the frame offset for DWARF offset(N) and val_offset(N) rules are declared Dwarf_Unsigned (named dw_offset) even though all versions of DWARF (Section 6.4.1, page 172 in DWARF5, page 87 in DWARF2) have specified that N is a signed value. For twos-complement machines (who has another integer format now?) simply casting the value to Dwarf_Signed works. But it's misleading and ugly. We wish to design a better way. But we'll keep the existing for source and binary compatibility.
We have revised the library internals (in 0.7.1, this is a second try in this Frame code) to use private structures and we then transfer, carefully, to the public structs for the functions requiring the public Frame structs.
As of version 0.7.1 the library does all its internal calculations with the N as a Dwarf_Signed, Records register numbers as Dwarf_Unsigned, and adds protection against overflow in addition and multiplication for offset(N) and val_offset(N) calculation. Because the library checks the calculations one is guaranteed that using values from dwarf_expand_frame_instructions() (if it returns DW_DLV_OK) one can do any necessary multiply of offset with data_align or code_align and never see overflow.
We are considering removing all use of libelf from libdwarfp and dwarfgen and instead coding that into libdwarfp. This will simplify configure, cmake, and meson build scripts and remove many ifdef instances from dwarfgen and test programs. It will also enable running libdwarf-regressiontests anywhere. This work is ongoing but is not public.
It would eliminate a lot of fuss in configure/cmake/mason as the functions will always be there. Current libdwarfp and dwarfgen use a grand total of 9 functions related to libelf. Not much to deal with.
Released version 0.7.0 20 May 2023. (Yesterday's issue is fixed in the release).
A new oss fuzz issue received today. May need to delay the release a bit (See DW202305-003 for details).
We are updating the planned release date to be 20 May 2023 because there are no known issues and all the testing has been done. So why wait till June?
We have specific improvements to libdwarf in mind to tackle after the release. We think the API is stable.
libdwarf-code is now registered at
OpenSSF Best Practices and
has the badge:
Thanks to Mateusz Janda for pointing out a mistake in the code noticing Elf extended, the test of e_shnum was not quite correct. The relevant elf document is refspecs.linuxfoundation.org/elf/gabi4+/contents.html
We've set the next release, 0.7.0 to be 1 June 2023.
As of today the main branch on github for libdwarf-code is "main". The libdwarf-regressiontests branch name is also "main" now.
The branch name 'master' has always bothered me. Github made it easy to switch names and suggested 'main'. The concept here is inclusive naming.
To update a local clone do:
git branch -m master main git fetch origin git branch -u origin/main main git remote set-head origin -a Optionally, run the following command to remove tracking references to the old branch name. git remote prune origin
The following link shows those same github branch renaming instructions. https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch
Thanks to aguemad (github) for providing a testcase showing libdwarf could not read objects using the extension mechanism to allow more than 64K sections as section count and as the section index to the section strings section. This is described in the 'man 5 elf' page in Linux. Note that the gnu compiler and linker (and apparently llvm, but I have no example confirming llvm-ld) emit a slightly different output than that man page calls for. Now libdwarf (and the readelfobj project object reader reader) can read the 32bit section count/number whether the man page version or the compiler/linker version.
Unfortunately the existence of 32bit section numbers required an incompatible change to libdwarf.h meaning a few people calling dwarf_object_init_b() and using struct Dwarf_Obj_Access_Interface_a_s (and all code using struct Dwarf_Obj_Access_Methods_a_s) may get a compiler error and will then need to alter their code.
March has been a busy month. Some volunteers have been creating tests of libdwarf and dwarfdump that involve improper command line use or fuzzed object files. Thanks to everyone named here. All 57 of these have been fixed and the fixes are on github.
Youngseok Choi is developing a fuzzer and reported various problems with dwarfdump crashes (due to option misuse and more) and library crashes (corrupted object files). Imaginative testing!
David Korczynski, Artur Gartvikh, Maksym Adamantis, and Viktor Sytnik created a really useful set of small test programs (see the fuzz directory in libdwarf-code and ossfuzz* directories in libdwarf-regressiontests) and arranged for fuzzers to use a large set of smallish binary objects as a starting point. It is so cool that the test programs do so many things that use the library in ways and in call sequences nobody else would do, but which are all allowed and should work, at least they should not crash libdwarf or get the library into an infinite loop. The ongoing fuzzing will no doubt find more issues, but already this has been really helpful and resulted in many improvements in the library.
Pedro Navarro noticed that libdwarf could not successfully read object sections more tham 2GB long because he had such sections. The 'read' man page on Linux has documented that limitation for a long time, but we were unaware. The fix is simple: make the read a small loop reading at most just under 2GB each time. Bug DW202303-057.
Two functions in the API for access to .debug_names (DWARF5) were never tested and it turns out one is slow, ineffective, and hard to use and the other was always broken and both are unnecessary. So dwarf_dnames_abbrev_by_code() and dwarf_dnames_abbrev_form_by_index() have been removed from the API. Fast and functional replacements already in the API will be mentioned in "Recent Changes" in the documentation. Due to the function removal the next release will be 0.7.0 (at the moment this change is only in a local branch, not on github).
We are now down to five functions in the API left untested. We are working on getting that count down to zero as soon as possible.
Preliminary 0.6.1 work happening in a local branch for the next few days. Merged to master (now main) in March. Using gcov/lcov properly means gcov points out clearly that seventeen libdwarf functions are not being called. So now we are working on ensuring those are tested.
Releasing 0.6.0 today.
Thanks to Ludovic Chopin for pointing out that CMakeLists.txt (in the base source directory) had check_c_source_runs where only check_c_source_compiles was really necessary. check_c_source_runs did not work in a cross-compile.
Building on Alpine Linux with -fsanitize gets an error around line 320 due to a false-positive error detector on !strcmp(postzprefix,targname+DPREFIXLEN). The false positive has caused build problems in various projects (including the Linux Kernel) and we were unable to find a small example with the problem. Other people saw dependencies on other gcc options. For protection of the build --enable-wall now adds -Wnostringop-overread, which does not prevent the warning but does let the build continue normally. Note that -fsanitize on Alpine requires 'apk add gcompat' and you will surely need to uncomment the community line for your release in /etc/apk/repositories for the add to work. And yet we still cannot get apps with -fsanitize to link.
Plan release of 0.6.0 on 20 February 2023. We'll be doing a lot of testing in the next days.
The dwarf_offset_list() function was behaving badly on 32bit-pointer systems. Fixed now on github.
Now a 'cmake install' creates a libdwarf.pc file. It's probably better to use meson or configure instead though.
A new compiler is using DW_FORM_strx3 (new in DWARF5) and that revealed a serious error in libdwarf. dwarfbug DW202301-001 has details. Basically DW_FORM_strx3 and DW_FORM_addrx3 were treated as four bytes long in dwarf_util.c. This error mixed up libdwarf and could lead to a segmentation violation or other serious errors in libdwarf. So dwarfdump behaves badly too. Fixed in 0.6.0.
Thanks to Andy Postnikov for alerting us to compiler warnings due to implicitit casts (fixed Jan 15). And for pointing out that for zstd the package file name is 'libzstd' (fixed Jan 16).
Thanks to ccccmd (github issue #146) for pointing out that the dwarfdump -h text failed to mention the -f option (prints the .debug_frame section). Pushed the fix to github.
Pushed fixes to DW202212-001 dealing with newly-reported omissions in checking libelf and DWARF content..
Work has been delayed as from 4 January 2023 5PM PST to January 7, 2023 4PM PST our block near Burlingame CA had no electrical power. Equipment failure PG&E says. What they don't say is that the first 2 days of the outage they left the bad equipment in place giving us 30 volts instad of 120 volts AC -- a brownout. Yes, a couple electrical items failed as a result.
Now back to testing a fix to DW202212-001, received 28 December 2022, where a heavily fuzzed object file bypassed all existing checks and uncovered four serious omissions in Elf object file checking.
We are changing the API in 0.6.0. Fixing a 1993 library design mistake.
We are changing the API of dwarf_get_pubtypes(). Code not calling dwarf_get_pubtypes() will see no change in API.By changing the API of dwarf_get_pubtypes() we eliminate 600 source lines in libdwarf and 2700 source lines in dwarfdump while leaving the existing capabilities intact. We also eliminate 30 functions from the API, documentation, and libdwarf code.
The change for dwarf_get_pubtypes (and the SGI sections mentioned below) has the effect of introducing a binary and source incompatibilty for those calling dwarf_get_pubtypes().
With respect to the sections involved: DWARF2 had .debug_pubnames. Dwarf3 and DWARF4 added .debug_typenames. DWARF5 has neither of these.
Four additional non-standard DWARF sections with the same format as .debug_pubnames were defined an implemented by SGI in 1993. Those never caught on (neither gcc nor clang ever emitted them).
In dwarfdump nearly all the instances of UNUSEDARG have been removed, mostly involving removing arguments from dwarfdump-internal functions (speeding up dwarfdump). A very small number will remain as a result of the way the tsearch tree-walk function interface is designed.
Release 0.5.0 issued 22 November 2022 In the release libdwarf.h DW_LIBDWARF_VERSION_MINOR is defined as 4 and DW_LIBDWARF_VERSION_MICRO is defined as 2. Those are wrong, they should be 5 and 0.
Reading DWARF DIEs is now several percent faster. The changes are library internal and do not affect the library API at all.
dwarf_get_globals() now also returns equivalent data (equivalent to that in .debug_pubnames) from the DWARF4 .debug_names section.
The definition of semantic versioning uses the term backward compatibility, as applied to a library such as libdwarf, is a bit unclear to me. The question, as of a couple days ago, is whether 0.4.3 is sufficient or whether 0.5.0 is required. Reviewing semver.org I think 0.5.0 is appropriate.
Any caller that uses libdwarf 0.4.0 or later can substitute in 0.5.0 and the caller's code will work. As you see below, the returned list from dwarf_get_globals() in 0.5.0 might be more complete (and thus more useful).
We modified dwarf_get_globals() to return all Dwarf_Global records from the DWARF5 .debug_names section as well as the DWARF4 and earlier .debug_pubnames section. In a single call.
The API of that function and the functions accessing Dwarf_Global record details are unchanged. We added a new function dwarf_global_tag_number(Dwarf_Global dg) which returns the DW_TAG of the DIE involved (for a Dwarf_Global retrieved from the .debug_pubnames section it returns zero, meaning no TAG visible there).
Libdwarf now supports a new section-compression library in addition to libz: libzstd, which has faster compression. These libraries are optional, but without the one needed for a particular section it is impossible for libdwarf to read the compressed section. Currently llvm is working on implementing this compression format, the clang/llvm option is --compress-debug-sections=zstd according to Michael Larabel.
libdwarf 0.4.2 is now released. Thanks to Herman Narkaytis for suggestions in finding and dealing with leaks in dwarfdump and libdwarf. That effort is now complete and neither fsanitize or valgrind nor CoverityScan report any issues of leakage.
Some test sets that run with only N malloc allowed to succeed for a wide range of N >= 0 were created for this effort.
Pushed the fix for a dwarfgen leak defect (via CoverityScan) today. Neither libdwarf nor dwarfdump leak now, even when malloc fails early or...whenever. That is not an absolute guarantee, but it is what tests show. Error reporting when malloc fails is much better now in a number of small ways. The fix to ossfuzz issue 51183 (libdwarf leakage of GNU debuglink and buildid sections) is included.
In rare cases dealing with reading corrupted object files or the -h command dwarfdump could leak memory (not large amounts). And libdwarf could leak a one or two 40 byte records. These issues appear to have been fixed though testing of this is incomplete. So work is still going on, But all the usual tests pass.
Added a github action building and checking libdwarf in FreeBSD. Thanks to Herman Narkaytis helping make this happen.
A tentative release date for 0.4.2 has been set: 15 September 2022.
As of Today Work In Progress is a separate html page making it a little closer to a blog :-)
Reminder: the libdwarf documentation has been entirely rewritten using doxygen as of February 2022. It has contents tables and links to tie things together, so it's much easier to read and find things than previously. See libdwarf-code/doc/libdwarf.pdf in release 0.4.1 or in the github source. An on-line html version is www.prevanders.net/libdwarfdoc/index.html
Vulnerabilities DW202207-001 and DW202208-001 were reported and fixed. Their fuzzed (corrupted) object files could possibly crash libdwarf. A big thank you to David Korczynski and Han Zheng for reporting the vulnerabilities and providing concise test cases. (28 August 2022)
Some fixes to the test build scripts (configure,meson,cmake) enable 'make check' (or equivalent for all three supported build systems) to work properly in Linux,MacOS, and now MinGW64 msys2 (Windows).
Warnings reported by Windows Visual Studio C compiler and by CoverityScan have been fixed. One of the warnings even identified a bug in dwarf_tsearchhash.c, though it was not a bug that affected any libdwarf API calls. Thanks to Vincent Torri for reporting Visual Studio's warnings. (28 August 2022)
Released library version v0.4.1. Includes fix for DW202206-001, a bug in dwarf_form.c when reading a carefully corrupted object file. (25 June 2022)
Thanks to several people who have provided improvements to and detailed bug reports on libdwarf/dwarfdump for release 0.4.1. Heiko Becker, Ilfak Guilfanov, klueke, MaqiGod, pedronavf, Casper Sun, and major contributor Vincent Torri.
Casper Sun reported, 26 May 2022, a previously unknown vulnerability in libdwarf that can lead to a segmentation violation reading a carefully corrupted DWARF .debug_pubnames (or .debug_pubtypes) section, and provided such a corrupted object file for testing. DW202205-001 (Fixed 29 May 2022)
Now doing builds on MacOS through github. See README.md for details. (April 17,2022)
The latest release is 0.4.0 released 2022-04-12. dwarf_xu_header_free() renamed to dwarf_dealloc_xu_header(). dwarf_gdbindex_free() renamed to dwarf_dealloc_gdbindex(). dwarf_loc_head_c_dealloc renamed to dwarf_dealloc_loc_head_c(). The unused Dwarf_Error argument to dwarf_return_empty_pubnames() has been removed. This completes the argument changes and function renaming that needed to be done. (March 4, 2022)
It's come to our attention that building libdwarf.so on Windows with cmake does not get -DLIBDWARF_BUILD defined. In src/lib/libdwarf/CMakeLists.txt, around line 77 try:
- target_compile_options(${target} PRIVATE ${DW_FWALL})
+ target_compile_options(${target} PRIVATE -DLIBDWARF_BUILD
+ ${DW_FWALL})
(February 25, 2022)
libdwarf.pdf has been completely rewritten. A much easier to work with libdwarf.pdf is in release 0.3.4, created with doxygen. That content is available on the web as html. www.prevanders.net/libdwarfdoc (25 February 2022)
We will be working on 0.4.0 soonish. Release 0.4.0 will have four function spelling changes and one argument delete, all to correct old mistakes. (25 February 2022)
Thanks to Vincent Torri builds of much of the package can be done with meson. A particular configure build takes 48 seconds by the wall clock, but a meson build takes 3 seconds with the same build machine and C compiler. The meson commands are not yet well documented documented in README.md. (cmake and configure will be continue to be supported for the foreseeable future.) Thanks to Vasilly Prokopyev for noticing the PE reader code was doing one sanity check incorrectly. (20 February 2022)
Thanks to Vincent Torri for major clarification and rearrangement of the #include entries in dwarfdump. (28 January 2022)