The Art and Science in the Quest for Height

Sailing is a sport which has a lot to do with science while its participants often feel they are playing with a form of art. The truth is that a scientist who turns his hands to art or an artist who turns his mind to science, will make the best sailor.
Height on the wind while sailing on the beat, is a quest which many sailors leave to pure art, or pontificate about over percentage points for maximum depth in sails. In the cold reality of a race or passage voyage in hard weather, it is the helm and crew who stand informed of the physical variables and boat's defined characteristics, who are able to apply the art of sailing the boat to its' maximum height.
The Boat's Envelope and Rig Settings
There is a fundamental maximum speed and best progress to windward (VMG in a line plus good tacks and boat handeling) which a given design of boat has. For keel boats, the limit of max speed is closely related in a proportion to the length of the water line. Placing more power in sail area has diminishing returns.
In practice this has a direct effect because as you escape the waterline speed in a planing boat you are by definition able to sail higher such that you tie back into waterline speed : the 9er boats, 18s and I14s can mostly plane upwind in reasonable racing wind speeds, while in marginal upwind planing boats like the Tasar, it pays to reattach the quarter wave to the transom line or the wetted area exit point on the hull by pointing higher.
The same is true of displacement keel boats but to a far less dramatic effect. If you have a polar diagram for your boat with several plots then you will probably see that maximum boat speed increases as you get to 90' TWA. However take a look at what happens at the top end of the beat : there is the stall "hook" if they have bothered to plot sailing above optimal angle. There is then a peak which is good boat speed and highest angle achievable, while the curve seems to fall away with increasing wind angle : look carefully and you will find that the boat speed is increasing all the way to the beam reach. 
In other words, by applying purely tabular data for the TWS/TWA you can work out at least that you are sailing too low if you are exceeding the expected target speed for optimum pointing. With a good knowledge of sail trim and a rig adjusted for the conditions, you are lucky enough to have your own judge and jury of VMG : namely the jib tell tales, the birgee/windex and the top leech tell tale on the main. These not only tell you height is acceptable, they also tell you when to change course and tune the sheets. No computers involved, they show how the forces are resolving at key points in a completely graphical way.
So a log (speedometer) or good GPS with relaible SOG is a real aid to achieving good vmg, but no more so than a good rig setting for the day and the tell tales. Also many sailors just get used to "heel and feel" and have their eyes off the instruments because they can have a sub critical refresh rate for their readings, 6 seconds on older or cheaper models.
If your boat is a one design or well characterised by the designer in terms of rig tune, then getting the right rake, prebend and luff- sag for the expected wind and wave conditions is a text book issue. You can then factor in weight if you are carrying more gear or more deck "lard" for your won boat. For a one off or less well defined cruiser then it is all about experiment and an accurate log with a known refresh rate will help you tune the boat and note settings which produce best speed.
One very common fault which creates poor pointing (height avhieved) is a forestay which is adjusted too short, especially on a fractional rig where even a couple of centimeters extra on the forestay bottle screw can move the mast tip forward maybe as much as 10 cm on a 24 foot boat, taking it out of the sweet spot for balance and pointing in the boat. The top third of the mainsail determines the majority of your boats ability to point and maintain optimal speed, thus best VMG. (This is worth reading up on yourself in a book like Lawrie Smith's 'Sails') When the centre of effort of the top of the sail is forwarrd from its ideal which it is designed for, then the mainsail does not work as well in turning the boat uptowards the wind. Also too short a forestay may make your mainsail leech too tight and difficult to trim as it pulls it forward. Finally to the converse,  too long a forestay can either produce an inbalance called 'weather helm' where the boat fights the rudder to turn , actually to pivot, up into the wind,. Altnernatively it can reduce pointing by making the leech too slack, such that it falls open and is not driving the boat properly.

o
Be aware that when racing in an OD fleet, you should look at the TWS and choose headsails and reefing as per your own defined limits, and not follow the fleet. For example the Sigma 33 nationals i.n 1999 had one start where everyone was convinced they could hold the big genoa, and you could see boats ahead juost driving off sideways, ie making massive leeway as their keels "lost grip" .they should have been using their no 2 s, which are of course the most under used sail in their wardrobe due this " hold the No.1 bravado.
The Golden Rule: Speed Before Pointing
As you can read above i mention boat speed and good settings ten times more than "height" and that is because they are  pre-requisite for acheiving max height and VMG.
Speed is needed to attach the flow of air to the sails and just as importantly, to the keel, rudder and hull along the wetted area and water line. The sails on most sub 4oft boats will be working fine at 4 knts boat speed, whereas a traditional keel may take another knot to "fly". This is why you need to use the speed and acceleration of the "tight reach" or "just cracked off a bit" in order to get this flow, before you can then work up on the wind.
Conversely, sailing dinghies and short chord keel boats (where the keel is like a knife blade) often fly the keel quickly but also loose the attached, laminar flow quickly too if "pinched" or if the sail settings are wrong and stall the speed quickly in lulls or gusts , or shake the boat in some way as the gusts hit.
In essence then you need to get the boat moving fast and then work up onto the wind, relying on sail settings and smooth helming with the feedback from the telltales, log and heel-feel confirming you are optimilising. Being "in the groove" needs this foreplay otherwise you may never achieve it, or you may not appreciate the feel of getting into it from lower speed stalls etc.
In a handicap fleet this has to then influence your tactics a lot because in a boat which is slow to build speed, your tacks will have to really decisive and fewer in number than your faster competition.
As you move with the fleet you should really be looking to compare your speed and pointing when you know you are in the groove at max height. For further feedback,  Look for the boats nearest you and similar boats if not in OD, dont try to find the enemy in a faster rated boat, or a boat which has clearly hit a faster wind band or block ahead or to the side of your course.
Height while sailing in rougher seas, falls off such that you maybe have more boat speed, or power in the sails but also that you don't stall when you hit the bigger lumps and can in fact steer round them more easily.
Sailing in waves is a subject in its own right, suffice to say that you need to put more power in the sails low down, and fall off a little to sail with less resistance.