So I have recently taken over maintenance of an in-house Content Management System, and database optimization is not really my area of expertise.
Anyway a couple of things fell out to my eye when I was looking over the code.
The php code is a little bit "spaghetti with meatballs" with little to no comments.
But the biggest thing: As far as I can tell, the original programmer decided to forgo table joins entirely in the database code (both implicit and explicit).
For example here is the process to display a page column:
Fetch element list from the database
For each element call a subroutine to check display permissions and if successful, call another subroutine to fetch the element's html data.
Each of the subroutines effectively calls a separate query for each element. And the permissions step, I believe, involves querying two separate tables.
Performance isn't really a problem at the moment, and I wasn't asked to look into this. Although the page requests are a bit slow in my opinion.
Is it worth trying to rewrite the SQL stuff? I am thinking that the increase in maintainability will be worth it in the end, and that it will make things easier for me should scalability become an issue in the future.
Or is it not really as bad as I think? Maybe I am just overreacting. An expert opinion would be appreciated.
Refactoring is an important part of development process. Ignoring this fact means more problems in the future. The part of the problem is that not many managers understand the importance of continuous refactoring.
I'd recommend you to read "Refactoring to Patterns" by Joshua Kerievsky that has many good examples of how to safely change existing code by implementing new design approaches.
As of your question about SQL queries, it may or may not be the first thing you need to change.
If its about making your job in the future easier, I would start refactoring the spaghetti code before fixing the queries. Once you have a nice design, it should also be more straight forward how to integrate (and maybe even eleminate) some of those manual joins.
I think it would be worth rewriting while you have the time to do so. You don’t want to put it off until it becomes a problem. I say, let the database do what it does best. In this case, table joins would certainly perform better than multiple queries in a loop.
Your question is a bit too generic to give a good answer to and you haven't provided enough information to make educated guesses either, for your particular situtaion.
Things to take in to account:
Will the system grow over time? And how?
If there will be more users online, more preassure on the system - then there's cause for alarm. Systems with bad design does not scale well.
Bad code is one thing, but bad database- and general system design is worse. I think the key might be your comment about "next few years". If you intend to stick with the sytem for that long - then a serious look at the basics is a good idea. If your system might be up for replacement by something else (inhouse CMS's tend to be replaced) then you can patch along while shopping for the right thing.
But as your question stands - rewrite some of the questions, make sure your DB is normalized and refactor your code. Tell your boss it's the right thing to do if you're going to stick with the system.
Related
I have been coding php enterprise applications from many years but still i would like to take a feedback.
Usually when i get the functional specifications, i create a worksheet of how long it will take to finish the project. I don't have a team and i code start to end.
Almost in most of the big applications after reaching the midway, i figure out that the database should have been designed slightly in a different way. So i create or delete or modify few tables (sometimes more than few tables) for better optimization and data fetching.
So in my experience, i always have to do a rework somewhere in the mid way.
Below are the steps i follow for big applications.
Visualize the database. I prefer sketching them.
I have coded my own PHP MVC Framework and i use that.
Create unit tests (if required)
Do some optimization.
Code cleaning.
Deliver.
My question:
I was never taught programming. I learnt everything through mistakes and experiences and ofcourse Stackoverflow. Am i following the right pattern ? Or is there a better approach to avoid rework. I am sorry, my question might sound noobish, but i am seriously looking for some helpful advice.
This ended up waaay too long for a comment...
Take StackOverflow with a grain of salt. Yes, there are some very smart, well-written answers, however there are also many cases of 'blind leading the blind'. You generally don't ask a question on SO if you know the answer, and if someone gets a terrible solution to their problem, but it works and they don't know any better, they can mark it as accepted like anyone else. There are a lot of people on this site in the mindset "if it's accepted, then it must be correct" and that will bite them in the ass eventually.
Find someone in your field that you personally respect and ask them to look over your DB design/methodology in detail rather than posing vague questions on SO. Your process sounds just fine in the broad strokes you've outlined, but who knows what kinds of devils are hiding in the details.
If there are certain small portions of code you'd still like to be reviewed by internet strangers you can try CodeReview.StackExchange.com. CR questions on SO get closed faster than you can blink.
I think your steps are good. You may consider posting some of your working code for review to get experienced programmers' opinions on design, best practices, etc. There are also chat rooms with lots of active users who are specialists in specific languages, if you don't have an expert on hand.
so I got a budget to clean up code that I've inherited from another programmer. the code is in really bad shape, there's lots of files not being used, there's lots of functions not being used. It actually looks like linear programming instead of object oriented. My issue is whether or not I should
a) Clean the code, restructure it, and try to delete all unused resources
b) start with a fresh framework (it's using codeigniter), and copy over the files that are needed to run
I like option b because it kind of reminds me of a new laptop from best buy. I can either spend the time removing all the bloatware or do a wipeout and create the system from scratch, which I always do. The only problem is that this time, there's a lot more involved then just creating a fresh system. Here are the pros and cons:
PROS
Clean system that is very easy to maintain
Don't need to go around searching for unused files
Easier to keep track of what I've done
CONS
Lots of things can break
Might miss required files
Might take longer
What do you think? Can you give me some of your pros and cons, and what you might do in a situation like this?
Update
A lot of people say that I am missing an important part, budget (time and money). We do have a decent budget, but my project manager is willing to go over if it ensures a more solid system with quicker turnaround time for new features. It's hard for me to quantify budget because you have not seen code, so giving hours won't help which is why I did not say anything about budget. I guess from your prospective, consider this a budget that can accommodate both solutions.
Often a hybrid approach works better. Keep the gold nuggets and toss the rest. Most likely there is some really effective code even in the worst project. Also, things that would be painful to rewrite and already work well, you might just clean those up a little.
It takes a little judgment to determine what to keep, but you can kind of have the best of both that way.
Cleaning someone else's code can be a nightmare. If you can actually choose and the stakeholders agree, I'd strongly recommend you to start over.
I had an experience like this last year and to this date, the software is still in pretty bad shape, it's almost impossible to track every mistake made by the programmers we inherited the code from and spend a LOT of time on support for hidden bugs and stuff.
I guess it somewhat depends on your time constraints and how intimately you'll need to know the project.
I just finished restructuring some code written by a math-majoring grad student. Bright guy, but he's not a programmer and the code was, as you said, very linear. Long story short, I rewrote about 90% of the code and took more time doing it than I would have liked. It would have been faster for me to start from scratch, using his code as a reference. Not only that, if I had planned on making as many changes as I had, I would have put more thought into the restructuring process. On the bright side, I now know all the code and concepts in this project very well.
On the other hand, if you don't plan on making many changes or having to maintain the code, maybe it's not worth the hassle. Get it to a usable state and tuck in back under the rug, so to speak.
My 2 cents...
This question reminds me of Joel's article I've read a while back.
The single worst strategic mistake
that any software company can make:
They decided to rewrite the code from
scratch... It’s harder to read code than
to write it.
http://www.joelonsoftware.com/articles/fog0000000069.html
This is my first time posting something like this. I'm a complete n00b at programming/php. I was trying to grasp all the tutorials out there and wanted to create a sort of framework to base projects off of for web db/applications. I came up with a few files for crud operations that I know must really be horrible code. I was wondering if I could get some advice in advancing this 'framework' to the next level. I'm not great at OO, but I understand the concepts, so I went more functional instead. I'm aware that the songs are gross, I don't need help recognizing that.
I would like some help on the database calls. I understand I should be using PDO (to help with injection), but I'm not sure how to convert this code to that structure. I also realize that at the least I should be using mysqli functions. Again, n00b.
I am also aware of validating and sanitizing data, but again, being so new to this, I don't know where to begin. Especially if I wish to keep it tight and small. The comments are funny from a programmers aspect, but hey, you work with what you got. I'm trying to grasp MVC and be able to tie in AJAX and templates/skins, but the crux of it is to work with databases.
I do like how there is one config file that works with all the other files, and how that results in small files for index, read and delete. The others get a little longer. Not that including files is makes me cool or anything.
I think I fail horribly at understanding MVC and structuring the project so that maintaining it and adapting it is easier. I also realize that there is a lot of html in those php files which is a no-no too. How do I fill in the gaps from what I do know (I know what every line of every file is doing) to what I should know.
The code can be found here (probably not sourceforge worthy, but I want to help others learn as I do).
https://sourceforge.net/projects/eleete/files/
Building such a thing takes a good amount of time, before you go doing so I would suggest looking into open source CMSs such as Wordpress to see if they would suit you better than putting in hours of work creating something yourself, the reason I say this is because to begin with you may think it won't be too hard to create one for yourself, but as you get into it you will realize there are many avenues that need to be explored in terms of security and features.
However, if you still want to build something the most important thing is to be creating reusable, and easily understandable code. For example if you want to create db entries for things like pages, you'll want to create functions to do so, to which you can pass variables such as titles etc. A good way to begin is to create functions for collecting data from your db, for example siteinfo() could be a function that returns an array full of site info. To make sure every file has access to such functions you will need a global file like functions.php for example which you will need to include in each of your other files.
Remember, building a framework is hard work, but if you do it properly and thouraghly you should increase your productivity a good amount - this is something you need to consider when building it, make sure you're not doing things in a roundabout way. Make it simple, make it reusable, and make it powerful.
First off, welcome to the community! I hope you find here people who are excited to bring your more into the programming fold. It's an exciting place to be.
I get the impression you're biting off quite a bit. I fear you're also under the assumption you can write the perfect program from the get-go, which is a flat-out myth no matter how long you've been in the industry.
As to the former observation, my recommendation would be to pick a simple problem and try to solve it. My favorite recommendation here are simple card games - blackjack, for instance. Dice games like craps also make for short, achievable, and interesting programs. The intent of these projects will be not to build a code framework as much as your skills in dissecting problems and turning them into a solution the computer can understand and solve for you- your code.
Starting small will keep you from getting overwhelmed early. Starting fun will keep you motivated through the inevitable snags that will tempt you to throw your computer out of a closed window.
All that said, we're still here to help! Specific questions to specific problems tend to get the best feedback here, as well as questions that demonstrate you've done at least some homework on your own in terms of trying to solve it. The Stack Overflow community will be happy to give you a hand up, but not a handout. Good luck!
You're going to be much better off using a clean, simple framework. As you're so new with this stuff, I'd recommend CodeIgniter for sheer simplicity and speed. It helps enforce the MVC framework you're not totally familiar with yet, has all the built-ins you're likely looking for, and has a decent community. Being a Zend guy, I should probably suggest that, but it's definitely not the easiest to get started with.
MVC and OO are concepts you really should learn well out of the gate before going forward. This tutorial rocks. There are dozens of MVC primers out there, but in essence it boils down to separating your data layer, view layer, and the like.
PDO is not necessary to be safe, you just need to escape your sql with Mysql_escape_string or the like. I'm actually not a fan of PDO due to the monkey wrenches it throws into debugging.
Since you're new, I'd also recommend Xdebug on your server for troubleshooting. It cuts down debug time immensely.
In my experience, unless you've had to fix the shortfalls in someone else's attempt at a framework, you won't really understand how to build one yourself.
One of the key qualities of any sort of API or framework is the concept of abstraction. In a nutshell, as you find yourself repeating code, look for where the repetition could be abstracted away. The trick about this when you're building a framework is to do it in an extensible, usable fashion so that you have pieces that build on each other. In other words, abstractions can hide other abstractions. For example, It's all very well to abstract away the job of assembling an SQL UPDATE statement from a list of fields, but why does the application page have to check the list of fields is correct? Put that behind another layer that can be taught what the valid fields are for a certain object.
Another key quality that comes up at this point is refactoring. The phrase "be prepared to throw the first one away" is sometimes bandied about in programming circles. What it really means is that you should always be open to the idea of re-writing code because you've thought of a better way to do it. Even if you have to rebuild the code that calls it.
I started coding recently, and this community has helped me a lot understanding many things which I was not aware of. However, many reputed coders instructed me of some patterns, the way I have to code and the way I shouldn't write codes. Although I accepted their suggestion with gratitude, there was many thing I couldn't understand.
I want your point of view to understand the few questions which has been running over my mind from the past few days.
MySQL
Why is it that many coders gave me thumbs down whenever I used the * in the select statement? Why did they suggest using entityname.tablename even if sometimes I want almost all the data from a table?
Is it okay if my code make a frequent trip to the database on a single page say about 5-8 request? To be more precise in a page I would want to update the value, insert the value, count the rows, sum up the values, and so on.
I once made almost all the select statement as a single query and one of the reputed user of this community suggested me not to do it that way instead suggested me to use a user defined function. (BTW, user defined function helped me a lot to clean my code and understand the error more quickly). What is your take on this?
Frameworks
When I started learning PHP I knew little about programming and more about web, although I had learned the popular computer languages like C, C++, .NET, Java etc. in my college. It was just the formal and theoretical subject which I learned and when I knew I wanted to be a web developer internet was my best friend and the community helped me out.
Now when I have started my journey of learning programming I have set some goals and aims myself, I want to be a Pro PHP Developer, I want to Master the HTML, JS, CSS, MySQL etc. My question here is
IS FRAMEWORK EVIL FOR LEARNERS LIKE ME?
MySQL
Using * is fine. When you start getting really complicated with MySQL queries - joining and comparing tables - then you want to look at using entityname.tablename just to keep yourself from getting confused.
The next question is too subjective. It depends on your server and the efficiency of your script. It also depends on how many people will be using the script. Obviously, as with anything, the less you use it the better. If you can do one sql query instead of 5 then do that, but if you're only going to have a couple of hundred people viewing your blog then I wouldn't worry too much. Its the same with functions. Obviously its much better to put everything into functions. This helps in the long run because you will only have to edit your script in one place to make changes. Lets put it this way - if you're copying and pasting code then you should be using functions. But then if your script is only 1 file, 200 lines long, then I wouldn't worry if you don't want to.
Frameworks
Frameworks are difficult to gauge the usefulness of. Obviously learning things like Zend or Mage are powerful frameworks that will help you to create much more efficient and complex web projects. However, for learners it may confuse you. I would say definitely not to try and learn them until you get your head completely around PHP. Hopefully then you will have a great enough understanding that you won't have a problem if you come across these. You miss the main point of a language if you learn a framework. For instance - you won't learn javascript if you just learn jquery. You'll learn a bit, but you'll never completely understand it.
Thats my take, but its a very subjective question.
Decent arguments to dislike select all in SQL: http://www.joelango.com/2007/04/30/why-you-should-never-use-select-star/
About number of queries: this is an issue of performance. The rule of thumb is to optimize when performance actually becomes a problem. If you are running a site that serves thousands of requests a minute you may need to start worrying. Otherwise, it just doesn't make a difference.
About frameworks: if you want to learn PHP at its foundation then avoid frameworks for now. Otherwise, if you really want to jump in and get things done, starting with a framework should be fine. For example, I don't know JavaScript itself but I work fine with JQuery (a JavaScript library/framework).
like Thomas Clayson says the use of * is fine for simple queries, but when you have complex join statements you should specify the fields and give the tables a name (like someTable as a) and use a.someField in order to organize your query.
When it comes to learning a specific language, frameworks are just tools to help developers get the job done in less time (including maintance, worst thing ever jaja) but if you are a starter what you really should look is to learn is what programming paradigm does the language support (object orientation, procedual or functional) and focus or learning those paradigm and the specific commands of the language because for example if you preffer object orientation class will always be class, function and procedures will always have the same squelet but the implementation differs in each language so if you learn the paradigm it will be easy to learn any language.
That my humble opinion hope will help, regards
While I agree with Thomas re: frameworks, I have to disagree, or at least expand, on what he says about MySQL.
While there is technically nothing wrong, in most cases, for using * in SELECT requests, expanding the column names makes the statement easier to understand and more self-documenting for other developers that might be in your code. They can look at the query and see what properties the row object should have. Aside from that, * is also slightly less efficient during the query. It's really nothing to worry about unless you only need a fraction of the columns available.
As for the multiple queries, it depends. If running 10 SELECT for specific items is faster than running 1 SELECT for multiple items and parsing them out, then run the 10 SELECTs. It's far better to run multiple small and fast queries than one large and slow query. Obviously, each application will be different.
On using * in a SELECT statement: While it is perfectly valid syntax, it is often recommended against for two reasons. One, speed; two: code quality. With *, it is not evident which rows you are requesting (and getting) and in which order; if the database is ALTERed and rows are added, you are getting these as well, and if the order of rows is changed, the results will be quite unexpected.
If you specify the rows you want, that results in safer and faster code.
That being said, you can still use * for convenience and some testing.
But for production code, please consider specifying the rows.
Making separate DB calls with SQL statements will slow down the processing a lot, since every one of these has to be sent, received, compiled and interpreted, results sent back - just to receive them again in another call.
stored procedures (or "user defined functions") are more efficient, and also much safer against attacks like injection. If you can make them without sacrificing flexibility or other aspects you need, do so.
But don't do that while you're still finding your way - Premature optimization is the root of all evil, says He.
Frameworks are quite useful, but indeed not a good idea for Learners / Beginners. They lead you to skip important parts of what you should be learning and limit your thinking. To judge the advantages and shortcomings of frameworks, to chose frameworks, and to overcome their flaws, you should have a solid knowledge of the basics first.
So learn your basics, and once you have a firm grip on them, you can start to use frameworks.
Please allow my intro to properly define the scope of my question:
I'm still very new to the programming world. This all started for me when I had an idea for a software program, but no programming experience. I ended up going the outsourcing route to get the program, and after almost a year, we do have it live and functioning.
This specific program is written with php and is 100% web-based. We're using lots of ajax, jQuery, etc.
Now a year into it, I have been learning and learning wherever I can (lots of learning here!!!) I'm mainly focusing on Java now to build up to Objective-C and the iPhone fun (probably like 99% of all other newbie programmers out there).
I'm really learning so much, and one of the biggest things I'm learning about is proper commenting and scalability.
I'm seeing now that this job we just finished is very sorely lacking in both those areas. I am wanting to add and build upon this program, and not only do I not have much experience, but I'm seeing that it's really hard for me to even get an idea about the functions without these comments...
So my question is-what is the best course of action to begin to pick up the pieces on this program? A full re-write is out of the question, and I don't think is needed.
I'm sure this is not the first time some newbie programmer, software developer has been down this path...what do others do here?
Is it common for a programmer to come into a project very far along and then "clean up" the mess in order to make things move forward productively?
If this is the wrong place for this question (and I understand it may well be) can someone point me to where this would be more appropriate?
Thanks!
Joel
We call it "refactoring" and it's an important part of programming.
First, you must have a rock-solid set of automated tests. Usually we have unit tests that we run with a unit testing framework.
http://www.testingtv.com/2009/09/24/test-driven-development-with-refactoring/
Then you can make changes and run the tests to confirm that nothing was broken by your changes.
In some cases, you have to "reverse engineer" the tests around the existing programs. This is not very difficult: you have to focus on the interfaces that are "external" or "major" or "significant".
Since you're reverse-engineering, it's hard -- at first -- to determine what should be tested (because it's an important external feature,) and what should not be tested (because it's an implementation detail.)
I'm really learning so much, and one of the biggest things I'm learning about is proper commenting and scalability.
First, I'm curious what you've learned about "proper commenting" as this varies drastically. For some, it's documenting every class and function. For others, it may be documenting every line of code or no code at all.
After having gone through some of the different phases above, I'm with Uncle Bob Martin who, in Clean Code, says that you document decisions, not what the code does. The code itself should be readable and not need documentation. By adding comments describing behavior, you've created duplication that will eventually become out of sync. Rather, the code should document itself. Using well-named functions and variables help describe exactly what the other intended. I'd highly recommend Clean Code for a full discussion of these concepts.
As for scalability, it's usually something that you want to build in. Scalability might be a function of good design, or a proper design for the requirements, but poor design will make scalability a nightmare.
I'm seeing now that this job we just finished is very sorely lacking in both those areas. I am wanting to add and build upon this program, and not only do I not have much experience, but I'm seeing that it's really hard for me to even get an idea about the functions without these comments...
I see this as an indicator of one of two things:
That the code isn't well written. Yeah, that's highly subjective. -OR-
That you don't yet fully understand everything you need to. -OR-
A little bit of both.
Writing good, intention-revealing code is hard and takes years of practice.
So my question is-what is the best course of action to begin to pick up the pieces on this program? A full re-write is out of the question, and I don't think is needed.
As other posters have mentioned, Refactoring. Refactoring is the process of changing code to improve readability and usability without changing functionality. Get a good book on refactoring, or start reading everything you can online. It's now a critical skill.
Is it common for a programmer to come into a project very far along and then "clean up" the mess in order to make things move forward productively?
Unfortunately it is. It takes a lot of diligence to avoid falling into this trap. Try to make your code a little bit better every day.
I don't know about this being the wrong place or not, but I'll answer as I can:
Is it common for a programmer to come into a project very far along and then "clean up" the mess in order to make things move forward productively?
Yes, in my experience this is very common. I have been doing contract work for over 10 years, and I can't count the number of times I've had to come in and clean up something hastily put together to either make it scale or to be able to add functionality onto it. This is especially common when you outsource the programming to another company, the incentive there is to get it working and out of the door as quickly as possible.
So my question is-what is the best course of action to begin to pick up the pieces on this program? A full re-write is out of the question, and I don't think is needed.
I don't know that there is a "good" answer to this question, the only thing I can tell you is to take it one method at a time and document what they do as you figure them out. If you still have access to the people that initially wrote the program you can ask them if they could give you documentation on the system, but if that was not included as part of the original work spec I doubt they are going to have any.
I'm really learning so much, and one of the biggest things I'm learning about is proper commenting and scalability.
As you have found on your own, proper commenting is important, I'm not convinced on the importance of building scalability in from the beginning, going by the YAGNI principle. I think that as any program grows it is going to go through growing pains, whether that is scalability or functionality. Could someone have built twitter from the start with the kind of scalability in mind that it currently needs? Possibly, but there is the very real possibility that it would flop.
Is it common for a programmer to come into a project very far along and then "clean up" the mess in order to make things move forward productively?
It's definitely common for pretty much EVERY programmer :)
Having said that, remember the IIABTFI principle. If It Ain't Broke, Don't Fix It.
Understanding how the program works and what the pieces are is useful.
Trying to improve it without a specific goal and a business purpose in mind is not.
The big question is how well is the program currently running meeting the needs of those that use it? While it may not be the best looking code, it does work which may mean that you end up doing 101 refactoring exercises around it to get enough of the basics down to make other changes.
While you may be able to ask the original writers of the program, this can be a possible sore spot if they think it is awesome and you think it is crap, for example. It is an idea and one that should be carefully analyzed a bit before one goes and ends up burning bridges because they think you can't appreciate their genius in what was done.
Often this aren't done in an optimal way and so as one learns better ways to do things, things are done in better ways. There is a limit to that of course, but I'd start with the idea that you have some refactoring lessons to help get the basics of the app under your belt and then start putting in enhancements and other stuff to see what was really done in the end.