I have written an Apache module in C. Under certain conditions, I'm able to have it to segfault, however i have no clue why. At this time, it may be my code, it may be the way in which I am producing this program, or maybe it's a bug within the OS library (the segfault happens throughout a phone call to dlopen()).

I have attempted running through GDB and Valgrind without results. GDB provides me with a backtrace in to the dlopen() system call that seems meaningless. In Valgrind, the bug really appears to vanish or at best become non-reproducible. However, I am a total novice if this involves these power tools.

I am a little a new comer to production quality C programming (I began on C a long time ago, but haven't labored appropriately by using it.) What's the easiest way that i can start learning the ropes of debugging programs? The other tools must i be looking into? To sum up, how can you learn how to tackle new bug challenges?

EDIT: Simply to clarify, I wish to thank Sydius's and dmckee's input. I'd reviewed Apache's guide and am fairly acquainted with dlopen (and dlsym and dlclose). My module works typically (it's at about 3k lines of code and, as lengthy when i don't activate that one section, things appear to operate all right.)

I suppose this is when my original question originates from - I'm not sure how to proceed next. I understand I've not used GDB and Valgrind for their full potential. I understand which i might not be producing using the exact right flags. But I am getting trouble determining more. I'm able to find beginner's guides that let me know things i know, and guy pages that let me know a lot more than I have to know however with no guidance.

This link might help: Apache Debugging Guide together with your specific problem. Knowledge about specific problems is among the how to improve within the general situation.

Regrettably the GNU tools aren't the very best, and my experience would be that the dynamic linker muddies the waters enormously. If you're able to get Apache to link statically together with your module which will allow gdb especially to do more dependably. I'm not sure how easy that's a great deal is dependent around the Apache build system.

It's worrisome although not shocking that you simply can't easily reproduce the bug with valgrind.

Regarding producing using the right flags, both valgrind and gdb provides you with far better information should you compile my way through sight with -g -O0. Don't think the claims around the gcc guy page that gcc -g -O is a good example it is not---even -O may cause variables within the source code to become removed through the optimizer.

I am certain debugging techniques have been in general language independent and there's no such think "C debugging".
There's lot of different tool that will help you discover simple problems like memory leak, or simply stupid mistakes within the code, some occasions it even can catch simple memory overruns. However for real difficult to find problems like problems came from from multitasking/interrupt, dma memory corruption the only real tool is the brain and well crafted code (with thinking ahead of time this code is going to be debugged). You'll find much more about planning your code to debugging here. It appears from Sydius publish that Apache already have a very good tracing mechanism in position, so only use it and add simalar for your code base.
In i'd state that another essential part of debugging is "don't assume/think". Base all of your steps on bare details, prove all of your assumption with 100% precision before you decide to making another step according to that assumption. Basing your debugging on assumption usually will take you to wrong direction.

Edit after Dave's clarification:

You next thing ought to be discover the littlest area of the code that create the issue. You sad when your disable certain section the module is loaded. simply make this as small can be done, remove/moke my way through the section until you will discover ideally one line which will make the module to not load. And once you find this line. it will likely be an exact time for you to begin using your mind :) Just be sure to 100% verify that this is actually the line.