FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory Fix

0

A few days ago our Ionic Framework 3 CI build stopped working when it tried to run the following command:

ionic build --prod

The error we were getting is the following:

Running app-scripts build: --prod
[12:36:35][Step 6/9] [12:36:35]  build prod started ... 
[12:36:35][Step 6/9] [12:36:35]  clean started ... 
[12:36:35][Step 6/9] [12:36:35]  clean finished in 2 ms 
[12:36:35][Step 6/9] [12:36:35]  copy started ... 
[12:36:36][Step 6/9] [12:36:36]  deeplinks started ... 
[12:36:36][Step 6/9] [12:36:36]  deeplinks finished in 639 ms 
[12:36:36][Step 6/9] [12:36:36]  ngc started ... 
[12:37:08][Step 6/9] [12:37:08]  ngc finished in 31.51 s 
[12:37:08][Step 6/9] [12:37:08]  preprocess started ... 
[12:37:08][Step 6/9] [12:37:08]  preprocess finished in 1 ms 
[12:37:08][Step 6/9] [12:37:08]  webpack started ... 
[12:37:09][Step 6/9] [12:37:09]  copy finished in 33.89 s 
[12:39:45][Step 6/9] 
[12:39:45][Step 6/9] FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
[12:39:45][Step 6/9] <--- Last few GCs --->
[12:39:45][Step 6/9]  1: node::Abort() [/usr/local/bin/node]
[12:39:45][Step 6/9] 
[12:39:45][Step 6/9] [10324:0x102801e00]   188080 ms: Mark-sweep 1413.3 (1493.9) -> 1413.3 (1477.9) MB, 3984.8 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3985 ms) last resort GC in old space requested
[12:39:45][Step 6/9]  2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
[12:39:45][Step 6/9] [10324:0x102801e00]   192147 ms: Mark-sweep 1413.3 (1477.9) -> 1413.3 (1477.9) MB, 4066.9 / 0.0 ms  last resort GC in old space requested
[12:39:45][Step 6/9]  3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
[12:39:45][Step 6/9] 
[12:39:45][Step 6/9] 
[12:39:45][Step 6/9]  4: v8::internal::Factory::NewStruct(v8::internal::InstanceType) [/usr/local/bin/node]
[12:39:45][Step 6/9] <--- JS stacktrace --->
[12:39:45][Step 6/9] 
[12:39:45][Step 6/9] ==== JS stack trace =========================================
[12:39:45][Step 6/9]  5: v8::internal::Factory::NewTuple3(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
[12:39:45][Step 6/9] 
[12:39:45][Step 6/9] Security context: 0x1c8c95225ee1 <JSObject>
[12:39:45][Step 6/9]  6: v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [/usr/local/bin/node]
[12:39:45][Step 6/9]  7: v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [/usr/local/bin/node]
[12:39:45][Step 6/9]     1: typeToTypeNodeHelper(aka typeToTypeNodeHelper) [/Users/mobile/Documents/buildAgent2/work/c6414cd03cae74de/node_modules/@angular-devkit/build-optimizer/node_modules/typescript/lib/typescript.js:~25875] [pc=0x169143ccb562](this=0x1c8ca3682311 <undefined>,type=0x1c8cb0e03f71 <TypeObject map = 0x1c8c588e4181>,context=0x1c8cb768bcb9 <Object map = 0x1c8c5888d151>)
[12:39:45][Step 6/9]  8: v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
[12:39:45][Step 6/9]     2: mapToTypeNodes(aka mapT...
[12:39:45][Step 6/9]  9: 0x16914338463d

Funny thing is, we made almost no changes to our app or our build process. So for the last 3 days, we’ve spent hours upon hours trying to figure out why our build was failing, backing out changes one by one to see which one caused the problem.

With no luck.

We updated node, npm, ionic and other packages to try and see if that was an issue was. Then we started to peg the versions of our npm packages and found the issue firebase. About a week ago Firebase released version 4.13.1, we confirmed that our package.json spec was not tied to a specific version:

{
"dependencies": {
"firebase": "^4.6.1",
.....
}
}

Unfortunately, that carrot (^) meant that npm would install any 4.x version of Firebase even if we were just trying to use 4.6.1. We removed the carrot and rebuilt without any issue. For more information on the special characters in the npm version strings visit the official documentation https://docs.npmjs.com/misc/semver.

If you’re encountering the FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory check to ensure your not using or accidentally using firebase 4.13.1 and switch to a lower version. You can force that lower version by only including the version number i.e. “4.6.1” in the string defined in your package.json file.

If you’re a First Responder, or know one, check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing free logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search & rescue, CERT, public safety, security and disaster relief organizations.

Development VM\Docker Server Configuration

So in my 2 previous blog posts, I went over why I’m building a custom built server in my house for development, storage (NAS) and testing (Docker). In this blog post I’m going to go over my setup and configuration of VMWare ESXi and some configuration & setup stuff I learned.

Frist, in my last post I mentioned that I was going to setup my RAID array as a RAID 5. After talking with some pro full-time IT\System Admin peeps I know, this is not the desired RAID configuration anymore. Basically, disks are now too large which pushes the total RAID size to the point where you’re taking a risk with your entire raid.  Drives have whats called a “non-recoverable read errors per bits read” rating, usually 10^14. This means that for every 12TB of bits read there may be 1 read that cannot happen and if that happens when you’re trying to rebuild your array because you dropped a disk, your entire array is hosed.

I’m already at that limit with 4 4TB WD Black drives in a RAID 5 configuration. Although I’ve rarely ever had an issue with WD drives (which is why I exclusively use them for spinning disks) I’ll be implementing a RAID 6 configuration for my home server. RAID 6 has 2 parity stripes which means if you did get an URE (Unrecoverable Read Error) the bit could be read from the other stripe.

IMG_6704

Screen shot above is converting from a RAID5 to RAID6 and adding 2 disks. This process took almost 5 full days with only 500GB of utilization on the array. So a LOT of time to do the rebuild, and a reason to have an external backup setup if you have anything important on that array. I’ll be keeping a spare disk

Once you get all your hardware installed into the case, it’ll be time to give it a boot, note this process takes a while. Then you’ll get to the LSI MegaRAID boot and you’ll have to configure your drives. This video (https://www.youtube.com/watch?v=KEmoon7eYf8) does a decent job of going over the setup of your raid using WebBIOS and this one (https://www.youtube.com/watch?v=v8ZfoEfGCgY) is also better to see what’s going on in the screen but there is only music.

IMG_6686

Note when you configure the RAID you have a few options for your Write Policy the default of which is “Write Through” this is the safest but has a performance impact. I used “Write Back” which is the highest performance\throughput mode but could lead to data loss if your system loses power during a write operation. For this, I have a UPS attached to the server and an agent VM setup to shut down the host safely during a prolonged power outage.

Once your array is created and healthy it’s time to install VMWare ESXi into your box. You can choose any path you want from here, but I recommend a bare metal hypervisor (one that doesn’t require a full host OS install) so you can allocate more resources to your guest OS’s. You can grab VMWare ESXi from here (https://www.vmware.com/products/vsphere-hypervisor.html) for free. It’ll start a trial, but you can convert to full time free (with a loss of some features) by installing the license. What what I can tell the only thing you loose is the ability to have a guest OS with more then 8 processors.

IMG_6721

Once you get ESXi installed, you should be prompted with a screen like the one above. You will need to ensure your IP address and network adapter are correct. You can press F2 to access the console and choose the correct adapter and set dynamic or static IP Address.

2018-04-25 11_31_35-Microsoft Edge

From here, all the work you will do in setting up ESXi will be via a web browser to your server’s IP address. You can manage, start, stop, adjust settings and monitor all aspects of your VMWare server from here. Note that not all browsers play nice with the ESXi WebUI and I recommend using Edge to play around with it.

In the next post I’ll go over some configuration and my VMWare setup.

If you’re a First Responder, or know one, check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing free logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search & rescue, CERT, public safety, security and disaster relief organizations.

The Build of my Development VM\Docker Server

In my previous post I talked about why I’m building a home lab server, the goals and starting info I was using. In this post I’m going to detail what I eventually got, the build itself and any gotchya’s that I encountered during that process.

My goal is to be able to have 6 Azure medium sized VM’s (4 core, 16GB of RAM) up and running all the time. That will save me $1,000 per month based on Pay-As-You-Go pricing. With still having some headroom left over for the host and other infrastructure.

Immediately I need to support the following VMs:

  • freeNAS Server for local sharing and backups
  • CyberPower UPS Agent
  • TeamCity build server and agent
  • Photon OS for DotNet Core and Docker development

Straight off the bat that’s 2 VM’s that would have cost me $330 a month. With the overall goal of adding up to 4 more. Resource allocation out of the gate will be:

Name CPU RAM
VM Host 1 500MB
CyberPower Agent 1 500MB
freeNAS 2 8GB
TeamCity 4 16GB
Docker Dev 4 16GB

Total

12 41GB

So now that I have my first set of system requirements, the load I’ll be placing on the system, it’s time to start buying the parts. As much as I can, I’m trying to buy the parts new and I was able to for most of the system, outside the processors which I got on eBay.

Part Link Cost
Intel Xeon E5-2670v2 CPU (x2) eBay $380 ($190\ea)
Samsung M393B1K70CH0-YH9 8GB PC3L-10600R RAM (x8) Amazon $190 ($23\ea)
Phanteks Enthoo Pro Full Tower Case Amazon $100
EVGA SuperNOVA 750 G2 PSU Amazon $120
Cooler Master Hyper RR-212E-20PK-R2 LED CPU FAN (x2) Amazon $60 ($30\ea)
PNY VCQFX370LP-PCIE-PB NVIDFIA Quadro FX 370 VIDEO Amazon $10
SAS9260-8I Kit Raid 8PORT w/ Cables Amazon $170
ASRock Motherboard LGA 2011 EP2C602-4L/D16 Amazon $380
Samsung 860 EVO 250GB SSD HDD Amazon $90
Asus 24x DVD-RW Amazon $20
WD 4 TB Black Performance HDD (x4) Amazon $760 ($190\ea)
CyberPower 1500VA UPS Amazon $140

Total

$2,420

To finish off the build above I’m missing 4 HDD’s (another $760) and 64GB of RAM (another $190). So the current total is short about $950 what the final total will be. To fully polish off the system is looks like I’ll be about $900 over budget.

IMG_6548

I also went expensive on the RAID HDD’s, opting for the faster WD Black Performance edition drives with 7200RPM and 128MB of Cache. Those HDD’s alone will turn into about 1/3 the systems overall cost. But that ensures that the VM’s and NAS will have fast access to the bulk of the storage on the system. You can save yourself a lot of money by going cheaper on the drives.

Some notes from this build:

  1. I also do not recommend the Cooler Master CPU coolers that I bought. Both fans mad a lot of noise and the coolers themselves are so large that if you don’t get a wide Full ATX case they won’t fit. I ended up buying a 2 pack of Corasir fans to replace the crap CM ones. Look for another LGA2011 fan or water cooler for your build.
  2. You do not need to buy the EVGA SuperNOVA 750 G2 PSU, you just need a PSU with 2 CPU cables (for the 2 procs). I think any modular PSU will be able to handle that, I know the SeaSonic ones I’ve been buying for years now all came with 2 CPU connectors.
  3. The motherboard on Amazon says it’s an ATX in the title, that’s 100% incorrect. It’s an SSI EEB server form factor board. You can buy ANY case you want, just as long as it supports Extended ATX (EATX) boards (12” x 13”).
  4. The Phanteks Enthoo Pro Full Tower is very wide at almost 9 1/2”, this is wider than my Cooler Master Stryker case which is pretty big. So ensure you plan accordingly with how much space you’re going to need vertically and horizontally.
  5. Never buy parts used in Crypto mining. This is why as best I could I got mine new. The CPU’s I checked with the seller to ensure they weren’t used in a miner environment. This is why I avoided CPU’s from Asia even though they were cheaper.
  6. I picked up a cheap (used) Quadro FX 370 so I could ensure I had a DVI port I could connect to my HDMI based KVM. I selected this card because I knew it would work with that generation of hardware, is low power and low profile. Note that if you pick it up you will need a special DMS-59 cable to convert it to DVI.
  7. I went with VMWare ESXi free edition, but finding out how to do that is a little hard, so here is a link to a VMWare KB article on how to license your trial edition for free.
  8. The RAID card I got from Amazon came with 2 the correct cables and the motherboard came with some for the SSD (where ESXi is installed) and the CDROM. There was no reason to buy additional cables (but ensure your RAID card will come with cables, as they are special 1 connector to 4 SATA ones).

IMG_6549

JP Toto’s excellent article on his build server (which I originally followed) has his out the door cost at $1,105 at 1/29/2017. But with the Crypto Mining frenzy, the prize on almost all components (CPU, RAM, PSU’s, Cooling, MB’s) have all risen. So keep an eye on prices to try and get better deals, you may have more luck at a local computer store getting good deals there as well.

If you’ve stumbled upon this article anytime in the future I highly recommend looking at V3, V4 or higher versions of the Xeon processes. With every version they get faster, more cores and more Cache. But you will need to buy DD4 RAM and a motherboard that supports that processor version, so just ensure you checking that compatibility. From what I read that ASRock MB that I bought will only support up to the v2 editions.

If you’re a First Responder, or know one, check out Resgrid which is a SaaS product utilizing Microsoft Azure, providing free logistics, management and communication tools to first responder organizations like volunteer fire departments, career fire departments, EMS, search & rescue, CERT, public safety, security and disaster relief organizations.

Go to Top