BMS reConnect #42: Factory vs. Field Installed Controllers
The Catch, Wi-fi jamming crooks, BACnet MS/TP troubleshooting, etc.
DISCLAIMER
All opinions are my own. Hit the subscribe button if you haven’t yet.
THE CATCH
Wednesday September 29th, 1954. Harlem, New York. The sun was shining on one of those perfect late-September days in the Northeast. The dog days of summer in the rearview, football weather still a few weeks away. Absolutely perfect.
If you were one of the millions of baseball fans in New York City that day, you were wishing you were one of the 52,751 fans that were able to rearrange their mid-week schedule, hop on the 8th Ave line (now the B&D) up to 155th St, and make it inside of the sold out Polo Grounds for the 1PM start of Game 1 of the 1954 World Series. The hometown New York Baseball Giants, National League pennant winners, were playing host to the Cleveland Indians, who had just set an American League record with 111 wins en route to a pennant of their own. For the first time in 6 years, the Giants’ cross-river rivals, the New York Yankees, were not playing in the Series. The Big Apple was all about the Giants that day, and the city was buzzing.
The game started with Cleveland scoring 2 in the first, answered by the Giants putting 2 on the board in the third. The score held at 2-2 for the next five innings. In the top of the 8th, Cleveland looked poised to break the stalemate. Giants starting pitcher Sal Maglie walked the first two batters of the inning leading to a dire situation as Cleveland slugger Vic Wertz took the plate. A pitching change was made, and lefty reliever Don Liddle was tasked with dealing to Wertz.
Wertz worked the count to 2-1 before blasting a ball to deep center field. According to one local sportswriter, “it would’ve been a home run in any other park, including Yellowstone”. But they weren’t playing at Yellowstone, and the long-forgotten Polo Grounds had one of the most cavernous outfields in all of baseball.
At the crack of the bat, the New York centerfielder wearing number 24—who was playing short to guard against the runner at 2nd base scoring—turned on a dime and sprinted towards centerfield. Displaying track star speed, the fielder closed fast on the descending trajectory of the ball. Seeing that the shot was just short of going yard, the outfielder, still in full sprint toward the warning track, extended his left hand above his head and made one of the most improbable, athletically impressive, iconic plays in the history of American sports. The Catch.
The ball was caught, the game was saved, and the Giants went on to sweep the Indians in 4 games, delivering their first championship in 20 years (and their last until 2010).
Willie Mays died on Wednesday at the age of 93. The 10th black player in the Major Leagues, 2x MVP Mays was one of the greatest players in baseball history. Playing CF in New York City in the 1950’s was rare air, with Hall of Famers Mickey Mantle and Duke Snider playing for the Yankees and Dodgers, respectively. Mays was the best of the bunch. When the Giants moved to San Francisco in 1958, he became the face of California baseball. From impoverished youth in 1930’s Alabama to being awarded the Presidential Medal of Freedom by Barack Obama, Mays was the living embodiment of all that is possible in America.
Sports has a way of elevating its stars into national icons. No living athlete was more iconic than The Say Hey Kid. Willie Mays. Rest in Peace.
FACTORY VS FIELD INSTALLED CONTROLLERS
Got ping’d after the last newsletter by a buddy of mine who is a PE, CEM, and commissioning agent extraordinaire:
His request: “You should do a blog post about the absolute sh*t show when it comes to integrating BAS controls with AHU factory controls. It’s always a mess and no one does it right and we always try and we get ignored until everyone’s tearing their hair out with testing failures during commissioning.”
Right on. Let’s do it. But what, exactly, my Cx friend, are the pain points you are dealing with? His response:
Design problems. “It's common for design sequences for larger equipment like AHUs or HW/CHW plants to have errors and gaps and ambiguities that will arise when the ATC contractor likely copies and pastes the sequence into their submittal and then stiffs the programmers with having to figure it out. Simpler equipment usually gets a copy/paste from past jobs and that often suffices but not always.”
Submittal documentation problems. “Typically no delineation between BMS and factory controls in the respective submittals, and they never clean it up when we ask them to. Factory controls always seem generic with critical options not selected.”
“The programmers often shoot from the hip to "get it to work" during install—or not— depending on which side of the bed they wake up on. They could figure it out well at this stage, and update their as-built sequences here, but they rarely do.”
“The fact that much controller programming is done offshore is part of the problem since those guys are also just shooting from the hip to some degree, and further disconnected from the end product.” Physical disconnection from the site conditions and local mechanical practices can lead to additional gaps.
“Cx tries to resolve this stuff during testing and make sure as-built sequences are accurate, but the challenges are daunting and the timeline is always crunched.”
My friend, you make some very good points here! I, too, have seen some nightmares with this type of application. Here are few thoughts I’ll share on the topic:
Common Problems
Compatibility Issues:
- Protocol Mismatch: The controls may use a different communication protocol (e.g., BACnet vs. Modbus vs. N2, etc) than the rest of the building automation system (BAS). The possibility also exists for the OEM controller to provide inadequate exposure of BACnet points to meet the requirements of the job.
- Non-Standard Interfaces: Some manufacturers use proprietary interfaces that are not easily integrated with existing BAS.
Configuration Challenges:
- Default Settings: Factory settings may not align with the specific operational requirements of the building.
- Custom Programming: Customizing the control logic to match site-specific needs can be difficult without detailed knowledge of the factory controls, which might be hard to come by if the OEM is using an “off-brand” controller or relying on off-shore engineering that may lack familiarity with local practices.
Coordination Problems:
- Commissioning: Proper commissioning can be hampered by a lack of coordination between the AHU manufacturer and the BAS integrator. Are all site-specific issues fully captured? Is the controller fully capable of executing the sequence of operations as intended by the engineer? What happens if there are mechanical changes…can the OEM controller accommodate them?
- Documentation: Incomplete or unclear documentation from the manufacturer can lead to misunderstandings and incorrect configurations. It can at times be difficult to get all of the proper engineering/installation information from some OEMs.
Support and Maintenance:
- Technical Support: Getting support for factory-mounted controls can be challenging if the manufacturer's service network is not robust.
- Spare Parts and Repairs: Sourcing spare parts or making repairs might require going through the manufacturer, causing delays.
Best Practices to Mitigate Issues
Early Involvement:
- Specify Requirements: Clearly specify your control requirements in the procurement documents. Identify these potential compatibility issues as early as possible.
- Pre-Approval: Get pre-approval of the control system from the BAS integrator before ordering.
Standardization:
- Protocol Standardization: Ensure that the controls use an open protocol that is compatible with your BAS. It is almost always BACnet these days, but that is not always the case.
- Uniform Control Logic: Request that the factory-mounted controls use standardized control logic that can be easily adjusted on-site. Confirm factory sequences align with design intent.
- Verify Compatibility Up-front: I.e., if the rest of the BMS be utilizing a Niagara-style BMS, ensure factory controls are also Niagara for best possible compatibility.
Detailed Documentation:
- Request Detailed Schematics: Ensure that the manufacturer provides detailed electrical and control schematics.
- Comprehensive Manuals: Obtain comprehensive operation and maintenance manuals for the controls.
Integrated Testing:
- Factory Acceptance Testing (FAT): Ensure FAT has been conducted to verify that the controls work as expected before shipment.
- On-Site Testing: Perform thorough testing and commissioning on-site to ensure seamless integration.
Training and Support:
- Training: Ensure that your team is trained on the specific factory controls.
- Service Agreement: Consider a service agreement with the manufacturer for ongoing support and maintenance.
Recommendations
- Collaborate with Manufacturers: Establish strong communication channels with the AHU manufacturers to address any integration concerns early in the project. But who’s job is it to foster this engagement?
- Engage Experienced Integrators: Work with experienced BAS integrators who have a track record of dealing with factory-mounted controls. Get them involved in the design process early.
- Use Modular Systems: Where possible, opt for modular systems that allow for easier customization and integration.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So…a lot to consider here! As is the case with many issues in life, one of the best practices is to nip the issue in the bud. Engage early and often during the design process. Promptly update the ATC vendor on any mechanical changes to the design.
If you have a sense of the overall type of BMS you will be installing, consider eliminating many of these issues by specifying field installed controllers, giving the ATC contractor more ownership of the outcome by allowing them to select their preferred control strategy up front.
What did I miss?
HOME SECURITY
Local law enforcement a few towns over issued an APB a few days ago about an emerging approach thieves are using to break into houses: Wi-fi jammers that will take out your internet so home security devices don’t work.
“NEW JERSEY - Police in one New Jersey town are warning that burglars are using Wi-Fi jamming devices to stop victims from using their cellphones to dial 911 before breaking into homes.
According to Florham Park police, a man was in the basement of his Lincoln Avenue home when he watched someone try to get in on his security camera. But suddenly, the camera wasn’t working.
When the would-be burglar noticed someone was home, he took off. The resident’s cellphone also wasn’t working, police say, so he ran outside, and someone called 911 for him.
"It’s like a cell phone blocker," said Robert Siciliano, CEO of Protect Now LLC. "They scramble signals, they make it very difficult for the signal to actually function properly."
Police are recommending residents consider hardwiring their security systems and have a landline as they keep investigating.”
None of this is very surprising to me. Just another thing to worry about. I’m thinking about reverting to more analog things in my life, in general. I probably won’t actually do that, but I’m thinking about it.
BACnet MS/TP TROUBLESHOOTING
Came across a good article the other day from Optigo Networks, a firm that specializes in OT network infrastructure solutions. A few snippets from that piece:
The 3 Most Common MS/TP Issues (And Fixes)
We’ve asked some of our fellow industry experts to weigh in on what they’ve consistently seen to be the most common issues and mistakes in MS/TP networks – and how to get them resolved without (too much) of a headache. We’ve come up with three main areas to look at – wiring, overloading, and device configurations.
1. Wiring
Most, if not all, of our experts agreed: When we’re talking about BACnet MS/TP troubleshooting, it’s almost always a wiring issue. MS/TP is essentially one long electrical wire, daisy-chained between devices. Joshua Durston, Controls Engineer at Modern Niagara reminds us, “MS/TP networks often go through many devices, making it easy to affect the whole network when a single device goes down.”
The most common issues, further described in the article, are:
Flipped polarity
Grounding issues
Bad cabling
Lack of end-of-line (EOL) resistors
2. Overload
It’s hard to overstate just how easy it is to overload an MS/TP network. MS/TP networks are old technology. Their data processing capacities are objectively tiny – 78 Kilobauds at the top end. These days, they are also frequently integrated into larger IP networks so servers, and monitoring software can centralize IoT management.
IP networks, on the other hand, are orders of magnitude larger and faster than an MS/TP network – 1Gpbs and beyond. That makes it easy for bleed-through from an IP network – say a typical discover (who-is) message on a complete system will completely flood a BACnet MS/TP router with more traffic than it can handle.
An IP network isn’t the only source that can lead to overload. Consider your MS/TP network itself:
Chaining together too many MS/TP devices on a single network can create long round-trips for data, leading to communication gaps and timeouts.
Poor “tuning” of your devices can also result in far too many requests for data and COV alerts than are needed, pushing even more data traffic that will need to wait to be processed.
Since MS/TP networks don’t keep a record of how many devices are present, it’s also easy to mistakenly set your Max Master settings. Too high and your network will continue to poll for devices that aren’t there. Too low and your system will simply skip over any devices beyond what you’ve told it to look for.
3. Device Configurations
Configuring devices on a BACnet MS/TP network is still largely a manual affair. While some newer devices allow for software commissioning, there’s still a fair bit of dip switches and cabling to deal with. Alongside this, MS/TP networks do not benefit from full duplex communication that BACnet/IP networks have. That combination inevitably leads to devices misbehaving due to their configuration. Here’s a couple to keep in mind.
Device configuration problems. One of the most common device configuration errors our experts come across is duplicate addressing. Because addresses for MS/TP network devices need to be manually assigned, it’s pretty easy to accidentally assign the same one twice.
Addressing gaps. While this isn’t a critical fault, keep in mind that your system will continue to poll for these “missing devices” in your network every 50 token cycles, adding delay on your network.
Conflicting baud rates can also be a common source of conflict as well. If some or part of your network is set to a different transmission speed, you run the risk of a communication breakdown.
If you or somebody you love are experiencing BACnet issues you can’t get to the bottom of, you could do a lot worse than reaching out to the folks at Optigo.
INDUSTRY EVENT
I am not currently at the Realcomm/IBCON event in Florida this week. I don’t know…I went to Florida in late June last year. I couldn’t take the heat.
But if you are not currently tuned into this group, it is one you should be aware of. Probably the premier coming-together of stakeholders interested in “smart buildings” and PropTech in the industry today. They do events throughout the year and provide a lot of resources. Definitely recommend checking them out!
ACOUSTIC JAMS
My tween daughter was blown away a few weeks ago when Fred told her that he had just seen Ed Sheeran at Boston Calling a few days earlier. She was super jelly.
And he’s super talented.
School’s finally officially out this week. When I’m not on the road, I work from my home office. From Sept-early June, it is a quiet and peaceful spot to get my office work done. From late June-late August? Pure chaos, as now there are an additional 4 people and a puppy to contend with. I wouldn’t have it any other way, but if we’re on a call and it sounds like I am in a bathroom, just know it was the only quiet place I could find.
Also: shoutout to my main man Joseph for turning 9 years old TODAY!!!
Hope you all have a great summer. Cheers!