Flanking is a gameplay mechanic in XCOM: Enemy Unknown.
Flanking is when a unit moves to a position that negates an enemy's cover and the associated bonuses. A flanked opponent will display a yellow shield icon next to their health. Flanking an enemy improves the soldier's chance to hit the enemy and to score a critical hit. However, the same is true of friendly combatants - if an enemy can get into a position to the side or behind an allied unit, they will be flanked, and be as vulnerable as a flanked enemy.
On Impossible difficulty setting, this vulnerability is the most apparent, as relatively low threat enemies such as Sectoids can easily one-shot XCOM soldiers in heavy cover if the alien units are positioned correctly.
Any soldier in cover at a corner is counted for the purpose of flanking as being able to fire round the corner itself. This can sometimes be seen by the game animations showing the soldier 'peeking' past the corner itself. Applied to doors, standing beside a door counts as firing from the door. Applied to trees standing behind a tree counts as being able to fire from either side of the tree. This is important because it means the soldier does not have to fully expose himself around a corner to get a flank shot. Any unit that does not use cover, such as S.H.I.V.s and MEC Troopers, cannot fire around corners this way.
Due to both the Critical Hit bonus damage awarded by a flanked shot, and the complete negation of enemy cover and other defensive bonuses, flanking is an effective tactic in taking down enemies with high health, good cover, or who have resisted all previous efforts of pacification.
When dealing with entrenched opponents, flanking generally requires that a friendly unit or units will move away from the bulk of the squad to initiate the flank. While this is a risky maneuver, any assaulting parties can be easily covered by either the Suppression ability of the Heavy and Support Classes, or (to a lesser extent) the Disabling Shot ability of the Sniper Class.
Take care when flanking over long distances. The further a squad member moves away from the sight radius of the rest of the squad reduces their ability to cover the flanker (this is not the case for Squadsight Snipers). It also increases the chance of stumbling on a previously undiscovered enemy patrol, leaving a small flanking party hopelessly outgunned. In such cases, destroying an enemy's cover can be a better approach.
Ghost Armor is a valuable tool to a dedicated flanker unit. The cloak ability allows a soldier to easily get into a flanking position undetected and exploit the cover negation to heavily damage or eliminate an entrenched enemy. In XCOM: Enemy Within, the same effect can be achieved with the Mimetic Skin Gene Mod or a Ghost Grenade.
To reduce the threat of an enemy attacking from an unexpected position, friendly soldiers should collectively cover as many possible flanking routes as possible with the Overwatch ability. Consider having a soldier positioned further back, whose Overwatch will only trigger when an enemy survives the first barrage of fire and continues charging. Being aware of all the enemies who are engaging you can greatly reduce the risk of an alien sneaking up on you; ideally, you should keep all enemies in sight at most times, but should you lose sight of one, consider keeping a running tally of enemies triggered and enemies killed. Be mindful of where the aliens were last seen, where they last attacked from, and observe the direction of the sound waves that occasionally occur during the start of the Alien Activity phase.
If all else fails, falling back to a position with better choke points is a good way to avoid the risk of flanking. The tight interior spaces of buildings, and their few ways in and out can allow you to easily cover all the possible ways in which an enemy could attack.
While on particular missions, aliens may enter the map in a position behind your squad. This is most commonly observed with Thin Men on various types of Council Missions, who will drop from above in unexpected positions, often flanking your squad. However, during this time, the enemy will only be on Overwatch, and cannot move or shoot freely until the next turn, also remaining uncovered in the meanwhile. Use this time to dispatch the enemy, or risk your squad suffering casualties from this flanking menace. If you cannot effectively engage the enemy that turn, use a Lightning Reflexes Assault soldier to break the Overwatch before moving the squad to a better position. Aliens entering the map in this fashion will be subjected to the reaction fire from any operative left on Overwatch with LoS of the target's "spawn" point.