MHFG's ¥400 billion system renewal project no walk in the park

Updated : 08.07.2016 / Category Economy


In an attempt to stride into the future, Mizuho Financial Group Inc. (MHFG)—a major Japanese megabank group—has undertaken a gigantic project to update its accounting computer system. However, the move is proving less of a walk in the park, and more of a death march.

In the software industry, the term "death march" is used to refer to harsh working conditions on such projects as software development, or desperate circumstances surrounding a project wherein engineers struggle with enormous workloads, with little chance of meeting deadlines.

It is said the system engineers presently developing MHFG's accounting computer system are literally working themselves to exhaustion.

"Working overnight and on days-off is common," said a system engineer sent from a leading midsized system integration company to a major vendor that was awarded the development project. "Each month, several engineers suffer from nervous breakdowns or depression and drop out of work. One day, a guy suddenly jumped up from his chair and screamed, 'How can you put up with a horrible job like this?' before kicking the door and leaving the room; he never came back. Needless to say, all the workers—even project managers—are worn out."

MHFG's accounts-related renewal and integration project now stands at a critical juncture. According to in-the-know sources, the project has already passed through the most difficult phase, but a string of troubles occurred during tests to assess the true worth of the just-completed system. If such techno-woes continue, the project may extend many months past the December deadline.

To compound concerns, some workers at subcontracted system integration companies are talking about the possibility of the project being "frozen" or "suspended."

In fact, some companies are said to have begun withdrawing or reducing the number of system engineers they have dispatched to the project. "Faced with an unprecedented shortage of system engineers, we can't afford to concentrate precious resources on this kind of death-march project," said an official of one of the companies. "It might go into deadlock at any moment."

'Pyramid of Khufu'
It was 2011 when MHFG announced its plan to renew and integrate its accounting computer systems—a cornerstone of daily operations. "We're going to build a foundation for the next 30 years," said a senior MHFG official at the time.

The move was triggered by two large-scale system failures—in April 2002 and March 2011—caused by the former Mizuho Bank. The second incident followed the Great East Japan Earthquake and revealed the fragility of Mizuho's systems, which failed to process data in dealing with a sharp increase in data associated with an upsurge in financial donations to accounts aimed at helping disaster-relief efforts. Furthermore, restoring the failed computer systems took a long time due to their fragmented organization, and the fact that nobody had a clear understanding of the larger, operational picture.

The current plan was designed based on lessons learned from theses failures. It differs from the conventional system integration method that consolidates different computer systems under one central system. Neither does it unify different systems by connecting them through relay computers. Under the present plan, a brand-new, next-generation computer system is being built from scratch, before three different accounting computer systems—viz., the former Mizuho Bank, the former Mizuho Corporate Bank and Mizuho Trust Bank—are merged into it. The plan is thus an attempt to realize total, simultaneous renewal and integration.

All the existing systems will be scrapped, with the exception of a small part of Mizuho Trust Bank's system. The systems to be junked include STEPS, a mainframe system manufactured by Fujitsu Ltd. used by Mizuho Bank since 1988, when it operated as (the now-defunct) Dai-Ichi Kangyo Bank. "This is an unprecedented undertaking in the banking industry," a Mizuho Bank source said.

Design work for the system started in spring 2013, with a unique design method called "Mizuho Service-Oriented Architecture" being introduced for the purpose. Under the method, applications are created for respective functions and services such as saving, financing and currency exchange. The applications are compiled in several independent components, which are then linked with one another via platforms implemented with necessary, common-use functions, such as those related to customer information files and the control of processing flow.

In the case of computer systems in which functions are repeatedly added over many years, applications cannot be separated because they are intimately connected with one another. Attempting to add new goods or change the service contents in such a system requires a lot of time and effort, due to the wide variety of parts that are affected. However, if application components can be configured with secure independence, new products can be introduced in a more flexible way, and related development costs can be significantly reduced.

The outstanding merit of such a system is that it stops applications from becoming bloated. This reduces computer bugs, and, even when trouble occurs, there is decreased risk to the whole system, meaning effects from problems can be localized and the system can recover more rapidly.

However, right from the outset, concern has been voiced over the project's future—not only from within rival banks, including Bank of Tokyo-Mitsubishi UFJ (BTMU) and Sumitomo Mitsui Banking Corp., but also from some members of MHFG. "The plan is reckless," said a source close to Mizuho Trust Bank.

The project has faced a mountain of difficult problems, such as securing sufficient human resources capable of managing it properly and enough system engineers with the necessary skills to development and integrate the computer systems.

BTMU had allowed two systems to coexist—those of the former Bank of Tokyo-Mitsubishi and the former UFJ Bank—via a hub connection. However, they were integrated in December 2008 under the former Tokyo-Mitsubishi system. For that project, about 6,000 system engineers were reportedly mobilized during the peak period. In term of man-hours, BTMU had the equivalent of 110,000 people working on it each month. The project cost a total of ¥250 billion and took two years from design to completion.

For the Mizuho project, the accounting computer system for Mizuho Bank alone requires 50 million steps to be worked on; if the systems of the former Mizuho Corporate Bank and the former Mizuho Trust Bank are added, that number jumps to 100 million steps. Put simply, MHFG not only has to construct one of the biggest systems in Japan, but must also integrate the old systems at the same time. The banking group is faced with "a very difficult technical stunt," according to an SMBC source.

Total investment for the project will be close to ¥400 billion, with about 8,000 system engineers needed during the peak period. Man-hours are expected to increase to 200,000 workers a month—nearly twice the figure of the BTMU project. "It's equivalent to the construction of a pyramid by the Pharaoh Khufu in ancient Egypt," an informed source said, expressing concern over how MHFG can hire and manage such a huge number of development workers.

Serious shortage of system engineers
The so-called multi-vendor system has been adopted for the project, a fact said to symbolize the extraordinary nature of the endeavor. The priority placed on politics among the former Dai-Ichi Kangyo Bank, the former Fuji Bank and the former Industrial Bank of Japan—the three institutions that merged to form MHFG—apparently is not the only reason for this.

The multi-vendor system comprises Fujitsu, IBM Japan, Hitachi—electronic firms that all have ties to the three banks—and NTT Data Corp., which has strength in developing systems for regional banks. The four companies were awarded the order jointly, as it was thought that even a major vendor effecting an all-out mobilization of its subcontracted system integrators and partner companies would likely face limits in terms of running such a gigantic project were it to win the order single-handedly. In a sense, "this method is an act of desperation," an MHFG source opined.

Once it got underway, the project soon met with problems, as predicted. As early as February 2014, management was forced to backpedal on the deadline, saying "designing the system took more time than expected." Initially, the project was due to finish by the end of March 2016. It was postponed for about one year, then rescheduled to end in December 2016.

According to sources close to Fujitsu, there was a serious shortage of system engineers around the time, due to increased investment in information technology by companies benefiting from the recovery of the Japanese economy on the back of "Abenomics"—Prime Minister Shinzo Abe's economic policy package. Consequently, at one point, even unskilled amateur engineers were assigned to the project. However, this concoction of skilled and unskilled workers only served to plunge the workplace into greater chaos.

"Actually, around that time, some vendor members proposed that the current project be scrapped and a new one be designed from the ground up, including specifications," recalled a senior official of Mizuho Bank, with a regretful tone. "However, our own systems section strongly objected to this. Accordingly, the bank decided to stick with the initial plan, regardless of the manpower required, meaning it could not turn back."

It is too late now for regrets, however. The project has already reached "a situation where there is no goal in sight," an informed source said.

Risk of increased trouble with current systems
The heaviest blow for MHFG is not the long delay in renewal and integration of the systems, but rather, an impediment caused by a plan to introduce various new services and measures to improve customer convenience scheduled to start under the new system, such as a 24-hour money-transfer service and an extension of Internet banking hours. Moreover, the effects of cost-reductions and a plan to streamline operations look set to fall significantly short of expectations.

For example, equipment costs were expected to decrease by "tens of billions of yen a year," according to a senior official of the group. "Freezing" or "suspension" of the project will apparently give rise to much larger impediments. Judging from the size of the system, if the project is restarted, "it's likely to take as least two to three years," said a senior official of a midsized system integration firm. MHFG's top management—including President Yasuhiro Sato—are likely increasingly fearful over the risk of system trouble due to delays with the renewal and integration. But until completion of the new system, the group must soldier on with the present systems, which are decrepit and fragmented, with a history of the two large-scale failures.

"If there's another system-related catastrophe, the Financial Services Agency will never forgive us," said a senior MHFG official. There's no doubt we'll be forced to close down."

"A similar fate awaits MHFG if a system failure occurs due to rushed, under-duress renewal and integration work, resulting in an incomplete system," according to an FSA source.

In its financial statement for fiscal 2015, which ended in March this year, MHFG reported net profits of ¥670.9 billion. This means Mizuho has risen to second place among the megabanks—surpassing Sumitomo Mitsui Financial Group, which only reported only ¥646.66 billion in net profits—for the first time in nine years, since fiscal 2007, which ended in March 2008. (Sumitomo Mitsui FG had been forced to write off huge impairment losses from stock holdings in an Indonesian business investment.)

This auspicious event is said to have considerably boosted the moral of those within MHFG, according to a source close to the group. However, MHFG is still faced with a very dangerous situation and must find a way to march to a different, more upbeat, tune.


This is a translation of an article from the July 2016 issue of Sentaku. The original article can be found here.