OK, I added the comma. What is worse is that it does not seem to work at all in the Samsung browser of my Android tablet. And I don't know how to debug there.

Another issue is how to handle the multi-path madness in the diagram I used as an example. I guess it is really the XBetza that sucks in this area. So perhaps I should extend XBetza with a special feature for it. Actually it doesn't really have to be an extension, just a clarification: the original Betza n modifier for non-jumping is ambiguous on oblique atoms. We could resolve that ambiguity by specifying that it stands for the set of lame moves along all geometrically shortest paths to the destination. That is, as many diagonal steps as the short dimension of the move, and the remainder orthogonal steps, permuted in any order. That would still be consistent with the meaning on orthogonal or diagonal atoms, where there would be only a single path that is geometrically shortest (i.e. where diagonal steps are considered longer than orthogonal steps). George Duke's Falcon then would simply become nCnZ.

Great!

There is definitely something wrong with how the vulture and the knightrider is evaluated. Check this game where the diagram's AI does silly exchanges:

{469049721} 1. f7 e9 2. e7 Yf9 3. d6 h9 4. h6 Ij9 5. Je4 e8 6. f8 Yxd3 7. Mxd3 Yxi3 8. Jxi3 Fxf8 9. exf8 Fe12

The game is Grand Apothecary chess alert!

Well, there definitely is something wrong. But I don't think it is the way the pieces are evaluated. (You can see that by clicking the header of the 'move' column in the piece table.) It seems there is something broken in the search.I will look into it.

If you make any progress with the nightrider sacrifices, please let me know!

Aurelian Florea wrote on 2022-10-09 UTC

There is definitely something wrong with how the vulture and the knightrider is evaluated. Check this game where the diagram's AI does silly exchanges:

{469049721} 1. f7 e9 2. e7 Yf9 3. d6 h9 4. h6 Ij9 5. Je4 e8 6. f8 Yxd3 7. Mxd3 Yxi3 8. Jxi3 Fxf8 9. exf8 Fe12

The game is Grand Apothecary chess alert!

I think this is simply classical horizon effect. The issue is that Ra2 is hanging (6... Vxa2). It is poisoned, though: after 7. Nb5 the Varan is trapped, and can at best be traded for a Cannon. If the program tries to capture the Rook early, it will see that it loses the Varan. But if it does not capture immediately, white can very easily 'rescue' the Rook, e.g. by playing 7. b5. So it tries to keep white busy, by forcing him to recapture for a number of moves (or lose a Kangaroo when he does not recapture, which is nearly as bad as losing a Rook) so he doesn't get the time to do something about the Rook 'threat'. Then, just before the search depth is exhausted, it captures the Rook. Because the refutation to that, Nb5, is a quiet move, it is not considered in quiescence search, and black thinks he has gained a Rook. Which is worth more than what he lost in the Nightrider-for-Kangaroo trades. So black thinks that sacrificing the Nightriders in the end will get him ahead, not realizing that this advantage will only last for one move.

Horizon effect has been plaguing computer chess from the beginning, and as far as I know was never really solved. Modern chess engines like Stockfish get around it by searching so deep (like 20-25 ply) that it is just not possible to delay trouble so much through sacrificing in a simple game like orthodox chess that it can be pushed over the horizon. But the Interactive Diagram does not search as fast as Stockfish, and Apothecary Chess is not as simple as orthodox Chess. I guess this is a case where "works as designed" is not the same as "works as desired"...

I have some unproven ideas for how to solve this. For those interested in the details of tree search: When a move that looked good at a certain search depth suddenly looks very bad after searching a ply deeper, trying the same move two ply later when searching for alternatives is very risky. Because the two ply that went before it have eaten away so much of the search depth that the remaining depth is insufficient for finding the refutation, while you do know there existed a refutation when you had not preceded it with those two ply. The move can only be good if thosce preceding two ply actually did something to solve the covert threat that apparently existed. But statistically this is very unlikely, especially in large variants.

So in a case where the move that was best in an N-ply search, but is dramatically bad in the (N+1)-ply search, it might be better to forbid that move to be played as 3rd move during the (N+1)-ply search and the (N+2)-ply search. Because the remaining depth for the 3rd ply would be only N-1 and N ply, which was not sufficient for recognizing that it was bad. But this will err in the other direction (but hopefully less frequently...): in some cases the moves played in the first 2 ply would interfere with the refutation, and by taking the pessimistic attitude that it won't, you won't be able to find such moves. But what can you do when you know you don't have enough depth to see a solution to a problem, and don't have the time to afford more depth?

Perhaps a more conservative approach would be to forbid playing a 'failing move' 2 ply later only in cases where these 2 ply sacrificed something. Like a Nightrider for a Kangaroo. If you could not safely capture the Rook before that, it is a reasonably safe bet you also cannot safely do it afterwards, and just assume that without actually trying. If you really want to be clever you could also remember the refutation to VxR (Nb5), and would allow VxR in the 3rd ply if the 1st or 2nd ply interfered with the refutation. (E.g. by capturing that Knight, or if white already moved the Knight away in the 2nd ply himself.)

To be honest I was fearing the horizon effect. Ok!... Don't worry too much about it! I think the AI does offer a simple demonstration of the game.

@ H.G.:

How important is the engine part (and thus the horizon effect) to a self-teaching program, such as the one that succeeded in defeating a world class human Go player? I would think that at first thought the horizon effect in Go is more of a problem to be dealt with than in FIDE chess, due to the large number of possible legal moves in a typical Go position.

That's unless there's something about Go that makes unforeseen moves often much less fatal than might be the case in FIDE chess, or less fatal than in more complex CVs such as the Apothecary CV complex (or e.g. my Sac Chess CV?). I'd thought that self-teaching programs meant the end of the world as far as a human ever staying best for a long time at any board and/or card/dice game, rather than self-teaching software + engine possibly combined with faster and faster hardware.

[P.S.: Note that Seirawan Chess may not have been a complex enough CV (not high enough branching factor?), as an engine surpasses top human players early on, I read somewhere; Arimaa had a pretty high branching factor, but fell to a machine over top humans after less than 2 decades (still not high enough branching factor? Some other reason?).]

https://www.chessvariants.com/diffobjective.dir/arimaa.html

Programs like that search very selectively suffer less from horizon effect, because they do not have a sharply defined horizon. The positions in the leaves of their search tree are evaluated by a huge neural network, which is trained on millions of positions. And it is unavoidable that this training also makes it understand a lot of simple tactics by pattern recognition. Such tactics can then never be pushed over the horizon.

Thanks H.G.

In Go there's a lot of board cells (points) but only one piece type (though dropped at least to start with). In a complex CV like Apothecary there's lots of cells (AND a ton of piece types, AND the pieces can move on the board, unlike for Go). That may make for a significantly more overwhelming number of simple patterns to somehow store for such a program than is the case for Go, but I wouldn't know if it would bother a self-teaching program enough.

Quick question: Does the diagram AI take into consideration the highlight restrictions from the BadZone function? I was thinking about reviving Suzumu Shogi and adding a rule forbidding pieces that burn from burning each other, and I have successfully defined a BadZone function to enforce it in the move highlights, but I worry that it won't be take into consideration by the AI. I tested it on my own interactive diagram, and it seems to work.

Yes, the AI uses BadZone() and WeirdPromotion().

BTW, I think the Diagram supports a parameter to make pieces 'relatively iron', meaning they cannot capture each other. I needed that for Tengu Dai Shogi. I don't know if that would be suitable for your case; if it is you would not need to define  a BadZone to handle that.

For some reason, when promoChoice is set to + for Shogi-style promotions, the piece names in the table don't show the promoted version when clicked a second time.

Edit: This is only true for the table that shows the selected piece.

I am not sure I understand this. It used to be that when you clicked the name in the single-row table for the selected piece, the Betza move description was changed to the text 'promotes to ...'. This was indeed broken. (It did not survive renaming the id of that table cell to include the diagram number.) Did you mean that? This was never done in any other table. I am even not sure anymore whether this is a desirable feature, now there are so many other ways to call up move diagrams. Anyway, I fixed that now.

I also fixed the problem that user-supplied scripts for functions like WeirdPromotion() would be used by all Diagrams on the page where they occurred. The new way to supply such functions uses the 'satellite' concept. If the Diagram definition contained satellite=xxx, the routines xxxPromotion(), xxxZone() and xxxShade()  now are used instead of WeirdPromotion(), BadZone() and Shade(). There still is fallback on the latter when the former are not found, for backward compatibility, but assigning a unique satellite name to each Diagram on the page should now prevent all confusion on what belongs to what.

BTW, I also allow xxxShade(x, y) to return a HTML color string, rather than just true or false to indicate whether the square is dark or light. If a string is returned the square will get that color for background. (E.g. returning '#FF0000' will make the square with that coordinates bright red.)

The problem was fixed when you fixed the piece promotion not showing. The extra satellite functionality for user defined functions is also quite nice.

I was dissatisfied with the capability of the AI to recognize delayed winning conditions such as baring or reaching a certain board square. At a given search depth this seemed out of character with the general level of play. The same problem was already present to a lesser degree with the recognition of checkmate.

The problem is that delayed winning conditions award a win only one move later, when it becomes certain that this move does not pre-empt the win through an immediately winning move (e.g. when the move to the target square put you in check) or a neutralizing move (such as counter baring to make it a draw). When it did not have enough depth left to do that 'verifying move', it would not award any extra bonus for meeting the winning condition.

To a human player this looked stupid, and rightly so. Testing whether the winning condition is met is very cheap, as it has already been caluclated by the last move that was done (capturing the last non-royal or reaching a target square), and the chances that the next move can pre-empt the win are not very large. I now made it such that the following move will always be searched, even in situations where this would exceed the requested depth. This can never cause a search explosion, as after that move the game will be finished anyway. It is just a matter of determining the result.

The checkmate situation is more tricky. It is important to know whether you can get out of check (i.e. whether the check is a checkmate), but if you can, the game does not end. So always searching moves that evade check does run a serious risk to get to infinite depth, because mutual perpetual checks (where the evasion also delivers a check on the opponent) cannot in general be excluced.

To compound the problem, it is far from trivial to determine whether you are in check. With pieces potentially moving in strange ways, the only reliable way is to just generate all opponent moves, to see if one captures your royal piece. If you would do that in every position, it would slow down the AI by a factor 2, as it now generates moves for both sides, instead of just for the side that was on move. And the AI already is so slow.

So I made a compromise: instead of generating all opponent moves, the AI now only generates opponent moves with the last-moved piece, to see if that delivered check. That means the AI remains comparatively blind to discovered checks (or hopper activations). But that is not so bad, because humans suffer from that too. So if it errs in this respect it will not look stupid, just weak. In addition, it only does this if there is at least half a ply of search depth left (it accounts depth in half-plies). Being in check then extends the search depth (if needed) by another half ply, so that there will be enough depth to consider all moves, to see if these are legal evasions, and recognize the checkmate if there are none. The fact that only half a ply extra is given is a guarantee that the search can never continue to infinite depth due to mutual perpetual checking.

I hope these changes will not slow down the AI too much. In any case, even at the default search depth of 2.5 ply it will now recognize it can deliver mate in 2, even when this is done by two non-capture checks. (Glitch is that it already announces checkmate on the first move. But that does not prevent you from playing on.)

For those intested in the remaining search algorithm:

There is no conventional Quiescence Search (which would extend capture moves indefinitely). Only recaptures of a higher-valued piece (LxH) or of an unprotected piece that became possible because of the opponent's last move (so recaptures and pin-punishment) always get a full-move extention, at any level of the tree. Captures that became possible through the previous move of the same side (e.g. with the piece that was moved then, or discovered threats), again LxH or of unprotected victim, similarly get extended wiith half a ply. That guarantees plunder raids cannot continue forever. When the victim of a recapture is protected, equal captures still get half a ply extension, HxL captures none.

Non-captures never get extended. Their search even gets reduced by a full ply if they did not become possible because of the previous two ply, but such reduction is revoked when they turn out to be good.

I added a new feature to the Diagram, for better supporting irregular promotions. Up to now this required a user-supplied JavaScript function WeirdPromotion(). In making Diagrams for many variants I noticed that this was often required, and mostly for a quite limited set of reasons. Like an irregular shape of the promotion zone (like in 'diagonal' variants). Or zone-dependent movement, like for Xiangqi Pawns, which are implemented as promotions to another piece type (possibly using the same image) when you enter another zone. Yet another case is promotion like in Grant Acedrex, where there is no promotion choice, but the piece you promote to is determined by the square where you promote.

These things can now all be done without programming through a new Diagram parameter morph. A line specifying this parameter must immediately follow the piece type it applies to. In principle the value for this parameter is a FEN-like representation of the board, the ranks separated by slashes (starting with the highest). Not all ranks have to be present, though; just enough to cover the (white) promotion zone.

Each rank can contain a number of piece IDs, one for each square on the rank. A piece of the type to which the morph parameter applies would then automatically change in the type represented by that ID when it lands on that square. When there is an astersisk instead of a piece ID, the normal promotion procedure (controlled by promoChoice) is triggered when such a piece lands on that square. (I.e., you can select a piece from the table.) When there is a period it means nothing happens on that square.

So for promotion to the piece that started on the promotion square in a variant with the FIDE setup you would have to specify morph=RNBQ*BNR , and no normal promotion (maxPromote=0). Promotion on e8 would then allow you to choose a piece. If the promotion zone was two ranks deep, you would also have to specify this for the forelast rank, like morph=RNBQ*BNR/RNBQ*BNR . There is a shortcut, though: if a rank contains nothing but a double quote, it will be considered a repeat of the previous rank. So RNBQ*BNR/" would suffice.

You also don't have to specify a piece for all squares of the rank: if you specify a lower number the string would be appended a few times to itself until it is long enough. So specifying morph=Q would mean you promote to Queen on all squares of the last rank. Specifying morph=QR would mean you promote to Queen on the light squares, and to Rook on the dark. For a piece that changes into a Rook on all light squares you could write morph=R./.R/R./.R/R./.R/R./.R . For promotions in a diagonal 7x7 variant where white plays from lower left to upper right you would write morph=******/......*/"/"/"/"/" .

[Edit] I now made it such that a piece ID mentioned in the morph string will never be interpreted as the piece itself. So that two forms of a piece with different move sets in different areas can use the same ID to morph into each other. In other cases of conflicting IDs the piece latest in the table would be assumed.

I have done a complete overhaul of the move-entry routine of the Interactive Diagram. It is now based on the move generator of the AI, which has a more elaborate system of move encoding than the old highlighting routines. In particular there is no longer any limit to the number of locust victims a move can have.

The new system works by generating a list of all moves for the side whose piece gets clicked, and then click by click delete the moves that do not correspond to the clicks that have been made so far. In principle all 'destructive' legs of the move have to be indicated in the order the piece traverses those (according to the XBetza move definition). So the first click serves to identify the origin, the next clicks all the locust squares, and finally the destination square. Hopper mounts or bends in a trajectory do not have to be clicked.

After a click has reduced the number of moves that still conform to the click sequence, the first not-yet-selected square of all these moves gets highlighted. If a square gets multiple highlights a click on it won't reduce the number of moves in the list to one, so it cannot be the final click. To indicate that, it is highlighted by a blue star (as in the old system). Even if only a single move would go over that square, but it is not the destination, it is highlighted this way, and the user is expected to click subsequent squares visited by the move until the destination. (There is in general no 'autocomplete of moves'.) Square that would finish the move entry when clicked, because it is only used as destination of a single move, are highlighted depending on the type of move (capture, non-capture, initial, castling...), as in the old system.

The system tries to be smart w.r.t. the click order, though. It recognizes moves that were generated as e.p. captures or castlings, and does not require you to click the e.p. victim or Rook as locust squares. For each move that is generated it assigns an order in which the relevant squares have to be clicked to select that move. For castlings it actually does define the third click (after King origin and destination) to identify the Rook, (which is essential to disambiguate a normal King move from a castling where the King steps only a single square). But castlings are an exception to the rule that moves are not auto-completed, so if there is no ambiguity it would not request the Rook click.

For locust captures it uses a heuristic to distinguish 'shooters' from 'tramplers'. If the XBetza definition of the move ends in a back-and-forth locust capture, that either does not consist of King steps, or the point from where it started was not reached by a King step, it is classified as a shooter. The click order is then modified so that the destination has to be clicked before the locust victim, and no click is needed to return to the destination. Tramplers keep using the standard click order, where they visite all locust squares before the destination gets clicked.

There is an additional issue: The AI's move format not only specifies squares that get mutated, but can also specify one or more squares where e.p. rights get created. Sometimes two moves to the same destination only differ in the e.p. rights they create (for a multi-path piece that can be e.p. captured). Betza's Doublemove Chess is an example of that. Such ambiguity is also recognized by the new move-entry system, and the e.p. square is in that case included in the click order (just before the destination). Normally e.p. squares would not have to be clicked.

The system seems to work now; I still have to integrate it with the AI and game-recording routines. It will not be used as the default move entry for the Diagram script until that is also done. But here already is a Diagram with a variety of pieces that show some of the possibilities:

The Forest Ox is a shooter, which optionally destroys an adjacent piece after his Knight move. The Valkyrie is a swapper, that can relocate a friently piece at its destination anywhere along its path. Both these pieces are from Odin's Rune Chess. The Dragon is the Lion Dog from the large Shogi variants; it can capture up to 3 pieces. The Lion is from Chu Shogi. Both these pieces are tramplers. The Double Knight can make two Knight moves in one turn, but only if both are non-captures. After a double move it can be e.p. captured (by Pawns, the only pieces that can do e.p. capture here). It also can do a single Knight move. The King has flexible castling, 0 to 4 steps.

I extended the functionality of the morph parameter. It can now not only be used for specifying irregular promotions / piece-type morphing, but also to specify inaccessible or goal areas for each piece type. For this a square in the board image would have to contain a ! (inaccessible), + (delayed win) or # (immediate win). A piece ID indicates forced promotion to that piece type, a * a promotion choice according to the rules specified with the promoChoice parameter.

I added a section to the article to describe the morph feature, also giving some examples of its use.

I hope triple captures besides those of the Lion Dog become a thing in the future. Perhaps by using two additional variables ex2 and ey2? It shouldn't be too hard, at least for updating the functions that use the variables for the move coords to use the and updating the code for highlights, though I could be wrong. If it becomes too complicated for the AI there's always the enableAI=0 option.

The new move-entry system that is based on the AI's move generator already can handle that (probably even up to 10 victims). It isn't used yet by the Diagram as default entry routine, but you can force its use by specifying newClick=1 .

The morph and captureMatrix parameters now work too (but the latter does not yet allow specification of antitrading modes other than iron).

That's perfect! Now that that's done, I can test a double-burning Suzumu Fire Demon.

Also, for some reason the Comments section is excessively narrow when viewed from the interactive diagram page.

One other question: how will the new move generation affect the pieceZone and piecePromotion functions (and other functions that use parameters representing move coordinates)? The new code does not appear to use the old variables that keep track of a move's coordinates. Instead a clicks array is used to store the coordinates for every clicked square in one variable. For obvious reasons, the old parameter list for pieceZone and piecePromotion wouldn't really work all that well.

Also, you should consider adding a section about the user-defined functions so people don't have to read the comments to figure out what they do.

I guess I forgot to have the callbacks for the Shogi promotion (on the yes/no message) call the new handler for finishing the move.

Should be fixed now.

As to documenting the WeirdPromotion / BadZone scripting stuff. So far I have refrained from doing that, because writing their own JavaScript would be beyond the capability of most prospective users (present company excluded), and dwelling too much on it in the description of the Diagram is more likely to discourage people from using the latter than to help them.

But there are other reasons as well: this part of the Diagram is pretty much experimental and still in development; I already changed the parameters used in the function calls several times, and even the current argument list of WeirdPromotion is not well suited for the new move-entry system, as it only a single e.p. square. I now also regret that WeirdPromotion and BadZone are separate functions. It would have been quite possible to have their tasks shared by a single function, which, when called with the full description of the move, would return whether the move should be rejected, result in game termination, change the piece type in a mandatory way, or offer promotion choice. And in the latter case it would be desirable if it could actually specify what can be chosen. E.g. by returning a string in the format used for promoChoice, overruling the latter.

And I actually consider it an embarassment that such scripting is needed at all. Ideally it should be possible to specify the rules solely through Diagram parameters. The newly introduced parameters morph and captureMatrix make a great step in that direction; these are very powerful features, which could satisfy the requirements for the large majority of Diagrams for which I originally had to write WeirdPromotion aor BadZone functions.