It is more than 50 years since the release of the film “Diamonds are Forever.” One key component of the movie plot was the kidnapping and subsequent impersonation of an extremely wealthy American businessman. Impersonation was made possible by a voice modulator that enabled his voice to be replicated in real time based on another character’s spoken words. Leaving aside the feasibility of creating the effect with the technology of the time - this was a James Bond movie after all – the deception was only possible because the businessman was known as a recluse who had seldom been seen by the people running his various business interests. Generative Artificial Intelligence, or Gen AI as it is known in the tech world, turns that paradigm on its head. It makes those with the highest public profile the most vulnerable to impersonation.
As the name implies, Gen AI creates new text, image, voice and videos based on an existing ‘body of work.’ The effectiveness of the output is dependent on the quality of the predictive algorithms, the amount of data available on which the algorithms can be trained, and the amount of money that is spent on training the models. Despite technologists’ hubristic wishes, the algorithms themselves are not where the magic lies. The real magic comes from access to data and the money to harness the computer power to train the models. This is why IT firms focused on winning in the Gen AI space, want unrestricted access to as much of the world’s digital archives as they can get, without having to pay for it. That is also why the current leaders in the Gen AI race are those firms that already have access to significant amounts of data, usually generated by their users.
Given the pace of progress in Gen AI and the amount of money being spent, it is almost certain that before the next US or UK general elections, it will be possible for almost anyone, including me, to create a robust, but entirely false video of any leading UK or American politician saying anything I want them to say. The cost to me will be insignificant. Most of the use will be spurious and intended to be amusing. However, there will be some situations where it is far from obvious that the video is a fake, which could be significantly damaging. There may really be an awful lot of genuinely ‘fake news.’ By definition leading politicians, along with social media ‘influencers’ and others with a high public profile, provide by some distance the most comprehensive and up to date digital archives. However, some leading businesspeople have also deliberately developed a much higher profile than was historically the case.
Of course, the political classes worry about the ways in which this could damage their chances of electoral success and/or be used by malicious actors. Whereas businessmen are only just beginning to see some of the negative elements of a high profile. Impersonation, with a view to completing a financial crime may be just one. Financial institutions traditionally drummed into new recruits the very first dictum as ‘know your customer.’ However the growth of online only banking, for businesses as well as individuals, has sought to make redundant the idea that bankers would meet their clients. Traditional banks are rushing down the path of minimizing direct customer contact in the interests of efficiency.
The most common impersonation frauds relate to Authorised Push Payment (APP). Criminals may impersonate sellers of goods, senior managers, family members, banks or romantic partners. In the UK alone APP fraud regularly costs banks more than $500 million per annum (which of course is in fact paid indirectly by their customers) in more than 100,000 cases each year. Similar frauds covering cryptocurrency involved losses of nearly $700 million in 2023. In 2024 in Luxembourg, it appears that more than $60 million was stolen from a charity through impersonation of the CEO, with the criminal then instructing an employee to make apparently genuine, but in fact fraudulent, payments. That is all happening without any use of AI. The good news is that losses may have peaked as the effects of better customer education and improved internal controls have made everyone less vulnerable, though at no small cost in terms of resources and reduced efficiency.
Gen AI threatens not only to reverse that trend but to move activity to a whole new level. Just recently a woman in France lost nearly $1 million having been convinced by fraudsters using, among other techniques, GenAI produced videos of news programmes, that she was in a relationship with Brad Pitt and was helping pay the cost of medical treatment. ‘You couldn’t make it up’ seemed to be the verdict of many people who assumed they personally would never be so stupid as to fall for such a scheme. This level of complacency is dangerous, as is the fact that the focus of payment regulators is on getting banks to reimburse victims quickly. The UK adopted new rules in 2024 that ensure victims will be refunded in full any amount less than £85,000 lost as a result of an APP crime. That seems almost certain to lead to a further spike in cases and losses.
As Gen AI content becomes ever more realistic and with banks looking to minimise in person interactions, the potential for losses will continue to grow. Phones and emails, even corporate ones, are vulnerable to hacking, which limits the safety provided by widely used 2FA (two factor authentication) protocols. Police are essentially powerless in most cases. The majority of money ends up in accounts of banks held outside the country of the victim. The police detection rate is lamentable in most countries, with criminals staying ahead of the victims, the institutions intended to protect the victims and the police.
There is natural complexity today in terms of who should be doing what to combat fraud (individuals, banks, payment systems, etc.). GenAI adds a new layer of complexity in terms of ways of impersonating key individuals that makes the job harder for all. Eventually new responsibilities and rules will have to be determined, supported by law and prosecution. How big the losses will have to get before that happens is anyone’s guess. However, it seems likely that the criminals have a long and profitable time ahead of them.
April 2025